There’s a strong pattern in ratfic of the protagonist “winning” by gaining the power to design a new world order from scratch—i.e. taking over the world. It’s a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it’s hard to unsee.
You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn’t have to be that bad! The writers chose to design them that way—I expect in significant part because that provides a narratively compelling backdrop for the thing they wanted to write about, which was their heroes taking over the world.
From my current perspective, this feels ominous in a kinda similar way as if I read a bunch of Marxist fiction from the 1800s and noticed that it always ended in a communist revolution.
Spoilers for a range of ratfic (unfortunately telling you which ratfic itself constitutes a spoiler):
EDIT: the spoiler tags seem not to be working on the images. Mods, can you help? For now I’ll add a bunch of space below. Expect many spoilers if you scroll down.
Unsong:
Worth the Candle:
The Waves Arisen:
r!Animorphs: The Reckoning
Project Lawful:
And some examples which (at least partially) subvert this trope:
Friendship is Optimal and Branches on the Tree of Time both involve world takeovers with some tension about how good or bad they are. In the former case, that tension is the main thrust of the story. In the latter case, it’s a bit more ambiguous: the ending sure seems pretty ominous, but it’s also exactly what the heroes were aiming for, and I’m not noticing much authorial prompting to question their reasoning.
The Metamorphosis of Prime Intellect is much more explicit about the “taking over the world” thing being bad. But it also came out in the 90s. Interesting that (to my knowledge) no rationalist has written a story which is so straightforward about the downsides of a well-intentioned world takeover.
Perhaps the direction of causation goes the other way? In the real world, it seems like we are headed towards a situation where 1-3 giant armies of superintelligent AIs transform the planet. Whoever controls those AIs, if anyone does, will be in a position analogous to all those fictional examples you give.
All the authors of all those works of fiction know this, or at least have been seriously entertaining this AI-power-concentration-hypothesis for years.
So, they are writing fictional analogies for the situation they expect to actually happen in real life. Except of course, since they are writing fiction, it has to have a happy ending.
Their expectation that it’s going to happen in real life is in fact correct unfortunately, as best as I can tell. Will the ending in real life be as happy as the endings in fiction? Of course not, but they shouldn’t be taken as implicitly claiming that, anymore than Tolkien should be taken as implicitly claiming that all this mechanization, industrialization, technology, etc. that he was reacting to was going to end happily ever after in real life thanks to the plucky heroism of some country bumpkins who end up destroying it all.
So, they are writing fictional analogies for the situation they expect to actually happen in real life. Except of course, since they are writing fiction, it has to have a happy ending.
Well, exactly what I’m disputing here is how happy the ending is. For example, imagine that all of these stories played out exactly the same, with the exact same amount of concentration of power. But instead of the heroes getting to use that power to reshape the world, the power instead goes to.… a random person off the street. I expect that these authors, if they were to write that kind of ending, would portray it as a maybe-happy-ish ending, but one that’s still pretty scary and uncertain.
And indeed, this is roughly how I’d describe the stories mentioned above where a mostly-aligned AI gets total power—Friendship is Optimal, Branches on the Tree of Time, and Metamorphosis of Prime Intellect. These stories really grapple with the sense of unease and tension that comes with almost everyone losing almost all their power.
Whereas when I look at the examples of ratfic above, the stance they’re taking seems to be “our heroes became dictators of the universe. This is a straightforwardly happy ending.” And indeed, on several occasions (maybe as many as half a dozen?) I’ve heard people describe the ending of Worth the Candle as one of the best utopias they can imagine. All of this really seems like a big ideological blind spot.
I don’t think I understand how you are objecting to my point. Straightforwardly happy endings are the norm in fiction. I think you are reading too much into their choice to make the ending straightforwardly happy.
Let’s think about, idk, the Star Wars series for comparison. In it, the plucky heroes go to war against the bad guys, kill loads of them in a series of military engagements, and ultimately emerge victorious with their faction the new rulers of the galaxy. This is presented as straightforwardly happy ending.
In fact lots of mainstream popular stories are of this pattern: The heroes defeat the bad guys in armed combat, kill loads of them, and then their faction has more political and military power afterwards and they all live happily and it’s great and uncomplicated.
Applying your reasoning to these popular stories, I guess you’d say: “Ominous. It seems like most Hollywood writers expect there to be a giant war in the future, possibly a civil war or series of assassinations. They are encouraging people to fight in this war / battle / revolution / assassination-series, and win, and they are portraying this as uncomplicatedly good.”
But I feel like there’s a better explanation for what’s going on: (a) Their audience likes fighting as a way of resolving conflicts, it’s consistently entertaining and interesting to most people, so that’s why so many movies conflicts resolve via combat, and (b) their endings are uncomplicatedly happy endings because again that’s what most audiences want, that’s the standard way to make a satisfying story.
I do actually think that the general trope of “the rebels winning is sufficient for a happy ending” is pretty indicative of poor ethical thinking.
But even Hollywood balks at their heroes ending up with literal godlike control of the world. For example (though I haven’t watched the series) my impression of the Avengers franchise is that they introduce a plot device (the infinity gauntlet) that gives its wielder godlike powers, the heroes use it specifically to defeat the bad guy and undo the damage he caused, and then they destroy the device.
In other words, they got to the exact point that ratfic heroes got to, and then their happy ending specifically involves them giving up the same kind of godlike power that ratfic heroes typically use to make themselves dictators of the universe.
Similarly for Superman: his happy endings involve him successfully using his godlike powers to beat the bad guys without changing the established world power structures basically at all. And I feel pretty confident that a big reason Superman doesn’t end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending.
tl;dr: there are many ways to make a story have a happy ending, and it’s quite indicative of the authors’ ethical and political views which endings they consider to be happy. The kind of endings that rationalists often portray as happy, mainstream scriptwriters seem to go out of their way to avoid.
And I feel pretty confident that a big reason Superman doesn’t end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending.
Yes. And I claim they’re wrong about that.
There’s lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help.
If I ended up with “ultimate power” somehow, by some mechanism that didn’t involve me taking on ultimate power for a specific narrow mandate, I think it is both ethically correct to use it to permanently end many (but probably not all) of those evils.
Oh, I think of “ending factory farming” as very far from “taking over the world”.
If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.
Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren’t laws against, like flying around them in a way which blows the buildings over).
This might escalate to the government trying to arrest him, and outright conflict, but honestly if Superman isn’t skillful enough to defuse that kind of thing, given his influence, then he doesn’t have much business imposing political changes on the world anyway. A politically unskilled and/or unvirtuous Superman trying to end factory farming could quite easily destabilize society in a way that is far worse long-term than letting factory farming end on whatever the natural counterfactual timeline is (without AI, maybe 20 or 30 years?)
Relatedly I’m increasingly coming to believe that this reasoning applies to Lincoln, and that we’d be in a much better position if he’d let the Confederacy secede and then imposed strong economic and moral pressure on them to end slavery.
Relatedly I’m increasingly coming to believe that this reasoning applies to Lincoln, and that we’d be in a much better position if he’d let the Confederacy secede and then imposed strong economic and moral pressure on them to end slavery.
Why? Because that was the beginning of the centralization of power with the federal government?
I think Alexander Hamilton was the beginning, but this seems like a big step. Vassar talks about how, from the civil war onwards, the American legal system needed to be optimized to rule a vassal state while also pretending that they weren’t ruling a vassal state. Can’t remember the specific examples he cited to me but I found it fairly compelling.
This might escalate to the government trying to arrest him, and outright conflict, but honestly if Superman isn’t skillful enough to defuse that kind of thing, given his influence, then he doesn’t have much business imposing political changes on the world anyway.
This seems like a weird set of claims to me.
First of all, yes, it seems very likely to escalate to conflict? Many nations would consider Superman’s property damage to their factories to be an act of war by a foreign power, and that he’s doing it in ways that technically circumvent existing laws is not going to make much of a difference. (That seems more annoying actually.)
He could try to negotiate for big, universal, changes to the food supply chain via diplomatic channels, backed by offers to trade various things he can provide, which generally seems like a better way to go about it. But if push comes to shove, I think he should back up his demands with effectively military intervention, if that could work. (Actually, my guess is that if he’s resorting to violence, he should mainly be relying on targeted assassinations.)
Also, the factory farming is not the only evil that our hypothetical superman should end. I think he should probably initiate regime changes in various authoritarian or dictatorial countries (eg North Korea, Eritrea, Afghanistan), or otherwise enable residents to leave the territory of those regimes. Again, he should attempt to negotiate for that diplomatically, first, but this is very clearly against the core interests of those regimes, and it seems very likely that he’s going to end up going to war with them.
Are you saying that if the diplomatic negotiations deteriorate to the point of military action, that means that our hypothetical superman has failed, and he would be better off retiring? Don’t existing legitimate countries go to war for far less noble reasons all the time?
I do think that having a superman around is extremely scary, and among other things, he should make the standards that he’s enforcing extremely clear and legible. He should make a point to never be capricious. He should write clear manifestos declaring the moral principles that he’s upholding and the logic behind them. And if his principles change for some reason, he should telegraph that very clearly. He should telegraph his actions with a lot of lead time, so that he’s a predictable agent to make plans around.
But ultimately he should declare:
Factory farms and oppressive regimes that people are not allowed to leave should not be allowed to exist. I am open to negotiations regarding the safest and stablest and overal best path toward ending those institutions. But that they are ending is not open to negotiation. After an offramp plan has been agreed to, or if the international community cannot reach a consensus on an offramp plan, I will oppose anyone maintaining these institutions, with superpowered military force.
If you don’t like that, tough noogies for you. Your national sovereignty is not more sacred than the sentient rights on which your actions are infringing.”
This is basically how I would want a hegemonic morally-motivated nation-state to behave.
What standard do you want to hold our superman / morally-motivated nation state to?
Are you saying that if the diplomatic negotiations deteriorate to the point of military action, that means that our hypothetical superman has failed, and he would be better off retiring? Don’t existing legitimate countries go to war for far less noble reasons all the time?
I indended this to refer to scenarios where the US itself (or other leading western powers) were taking military action against Superman. I care much less about whether he destabilizes North Korea or Eritrea or even countries similar to those but better-governed. But I care a lot about whether he destabilizes the countries I consider the best and most important ones.
Many nations would consider Superman’s property damage to their factories to be an act of war by a foreign power
Maybe. Or maybe they really wouldn’t want to pick a fight with Superman. Or maybe they would issue an angry press release then not do anything. In a setting where Superman holds basically all the cards in terms of physical force, most nations would try quite hard to defuse tensions with him (unless, as I discussed, he’s very unskilled).
I’m still not sure what standard you’re holding our hypothetical superman too. You just don’t want him to destabilize the countries that you consider most important?
If he overthrows the Communist Party of China, is he violating your standards?
If he forcibly institutes electoral reforms to make the US government more functional, is that violating your standards? Is the “forcibly” there a problem, and if it comes to using military force to push for electoral reform, he should hold off?
I claim that there are some people such that, if they were dictators of China, that would be much worse than the current situation. And there are some people such that, if they were dictators of China, that would be much better than the current situation. Which category a given person falls into depends a lot on their honesty, integrity, wisdom, ability to understand political dynamics, ability to resist manipulation, etc.
There are no particular limits I’d want to place on a sufficiently virtuous Superman. E.g. I want Superman to follow a policy that leads him to overthrow the government of China iff he is in the latter category. The big question is how Superman can gain justified confidence that he’s in the latter category, given that unvirtuous people are prone to a lot of self-deception. One way he can do it is by setting limits on his own behavior so that he can gain more evidence about what kind of person he is. E.g. maybe he thinks he’s really wise about politics—wise enough that him having control over US electoral policy is a good idea. If so, he should try to test that wisdom by implementing political change without using violence. If he starts telling you that he doesn’t need to pass such tests, because he’s already so confident that his plan is a good idea, then you should start getting worried.
In other words, when I think about a question like “should Superman forcibly institutes electoral reforms to make the US government more functional”, I expect that there are some ways to do this that are really good, and some ways to do this that are really bad. And the kinds of people who are capable of doing it in a really good way (given that they’re Superman) are also generally the kinds of people who wouldn’t need to use much force to make it happen (given that they’re Superman).
(Modulo, I have more uncertainty about how much force the wisest path entails when, one has hegemonic power. Certainly using military force to get your way has major costs, and so, taking those costs into account, I would expect the wiser courses of actions to be more peaceful, generally.)
Should I then summarize your criticism of ratfic protagonists as something like:
Trying to radically reform the world isn’t bad. The world does need reforming in many respects. But whether you are doing will do a good job at such a high stakes task depends on your personal virtue.
We can get evidence about how virtuous a person is by seeing how wisely and skillfully they comport themselves in lower-stakes situations where they don’t have all the power. Wise and virtuous people can generally make meaningful progress on their goals without needing ultimate power over everything, and without remaking the whole world in one shot.
Therefore, if a given person’s plan is to attain ultimate power, and only then use it to remake the world (instead of a more incremental process that doesn’t depend on centralizing power in their own hands) that’s a big red flag that even if they did end up with ultimate power, they wouldn’t be skilled or virtuous enough to use it well—they will likely make things much worse.
In general, ratfic protagonists tend to think that they already have all the virtue that they need to wield ultimate power, because they can see the inadequate equilibria in the world and can identify the better equilibria which could exist if only they had the power to make them so. They act as if most of the problem of wielding power is correctly identifying what to aim for, rather than procedural and personal questions of how to wield power well (so that you end up accomplishing your noble aims at all, and avoid causing a lot of harm along the way).
The more a person thinks that the thing that they need to make everything good is “more power”, the more concerned we should be that they are undercounting the importance of virtue and wisdom, and the more worried we should be if they actually ended up with ultimate power.
I’m also reminded of something that Val used to say: “power felt is power wasted”. Deploying a lot of force to make things go your way is very inefficient.
If you’re skilled, you should be able to get what you’re aiming for while deploying very little actual force (“speak softly and carry a big stick” for instance, but also using soft power and good leadership more generally). Someone with a little power and a lot of skill can often do as much (and with less collateral damage) as someone with a lot of power trying to muscle through.
Power is definitely useful, but the more you think that the thing you need to accomplish your aims, the more that indicates that you don’t have skill with using power efficiently.
I broadly agree with this comment too, though not as much as I agree with the other one.
Power felt can also be a kind of honesty—e.g. if a law is backed by force, then it’s often better for this to be unambiguous, so that people can track the actual landscape of power.
(Of course, being unambiguous about how much force backs up your laws can also be a kind of power move. I expect that there are ways to get the benefits of honesty without making it a power move, but I don’t have enough experience with this to be confident.)
In other words, I expect that the kind of inefficiency Val is talking about here is actually sometimes load-bearing for accountability.
Brandon Sanderson’s books have some interesting variants of this as well.
Spoilers for the Mistborn series (to the best of my recollection and with some consulting of the fandom wiki) (putting in collapsible for now, because I can’t get spoiler blocks to work with this new editor):
At the end of The Well of Ascension, Vin gets a hold of the power of Ruin (one of the Shards, roughly god-like entities of the universe). She knows that she can use it to eliminate lots of the world’s atrocities etc. Sazed, informed by a misty ghost, tells her that “it’s a trap” and she needs to release the power. Some other misty ghost appears out of nowhere and stabs Elend, nearly lethally, apparently to force Vin to accept the power, so that she can save her loved one. But she doesn’t do that, releasing the power, which turns out to free its prior weilder who is the bad guy and now goes on to destroy the world. He was also the spirit that told Sazed to stop Vin. Whereas the ghost that stabbed Elend was the Vessel of another shard, Preservation, a good guy (relative to Ruin, at least).
At the end of the next book, The Hero of Ages, Sazed accepts the powers of Ruin and Preservation, ascending to godhood and becoming Harmony. He approximately fixes the world, but then in the following series, it turns out that he’s not as powerful, as we might have predicted, and also the divine power that he has been wielding starts shaping his mind, so that he becomes more interested in things being Harmonious, than in “goodness”.
ETA: it’s plausibly relevant that Sazed, at the moment of Ascension, absorbed all the knowledge that had been stored for millennia in his Coppermind “amulets”, so that he could learn from the mistakes of the past generations and not screw things up as much.
IDK what the morale is supposed to be here, if any. “Sometimes you need to power-grab because otherwise someone else will power-grab in your place, but also beware because power corrupts and divine power corrupts in an ungodly way, so attaining instrumentally convergent goals is of limited value if it meddles with your utility function in unendorsed ways, be it due to some contingent peculiarity of your mental structure, or some more general fact of how minds work.”?
(There might also be something along those lines in other Sanderson cosmere books, but it’s been long since I’ve read any and I’m not up to date.)
ETA: Another relatedly interesting plot twist in Mistborn is that it turns out that the original bad guy, the Last Emperor, who gets killed at the end of the first book, imposed an oppressive regime in order to prevent humanity from Ruin. He also changed the planet for altruistic reasons that turned out to be misguided and hence harmful.
Relatedly, A Practical Guide to Evil is one of my favorite books/series, and grapples with the tension between trust and power very well. It’s one of the very few narratives I’ve seen written skillfully enough that the protagonist giving up power didn’t seem straightforwardly stupid to me (even when I was in a classic rationalist mindset).
It seems plausible that what you suggest is one significant contributor. Here’s one more thing that imo plausibly contributes significantly:
Most of these people are consequentialists, i.e. they think of ethics in terms of sth like designing a good spacetime block.[1] Like, when making a decision, you are making a decision as if standing outside the universe and choosing which of two spacetime blocks[2] is better. Given this view of ethics, it is very natural to imagine a future in which there actually is some guy that designs/chooses a good spacetime block, and it becomes somewhat less natural to imagine futures in which the spacetime block keeps getting “designed/chosen” in a messy way by all the messy stuff inside the spacetime block, with the designing/choosing and the being-valuable done by the same entities. A person who thinks in terms of duties or a person who thinks in terms of virtues would find it much less natural to have such a strong separation between the locus of moral-agent-hood and the locus of moral-patient-hood.
There’s a story that the reason why you have a lot of anime that goes in very weird sexual directions is that Japan is an incredibly sexually repressive society, so the Japanese channel their compressed libido towards art.
I don’t really believe this story, but I think the pattern it exemplifies is at play here.
Rats have very ambitious goals of fixing the world. They see that the world is in a pretty bad shape and demands fixing. But they can’t fix it. No one else can fix it either (nihil supernum, deep atheism), and even if they could, it would likely be bad, because their values are not yours (even deeper atheism). So you yearn for a world in which you smash those limitations. “Power” (or a specific kind of it) is the thing you need, and its value is not bounded, so the threshold of a superstimulus is largely your imagination, perhaps constrained by some ontological assumptions. That’s why this theme recurs through ratfic so much.
This seems somewhat obvious to me[1] and so I am somewhat surprised that people in the comments mostly seem to explain it in terms of game-theoretic (or other) realism or people having consequentialist-ish views, etc.
First of all, there might exist cases when a god with different values is preferable to the current world state (e.g. if the world is clearly heading towards self-destruction which would terrify even the god). Additionally, I doubt that the theme of gaining power and fixing the world recurs just through ratfic and not through fiction whose authors display some bias which I cannot describe more precisely than “finding it hard to restrain themselves.” Finally, Max Harms has been trying to construct an agent whose sole goal is being corrigible and even defined power for the agent to optimize in a way which I suspect to be transformable into the Natural Abstract Goodness.
First of all, there might exist cases when a god with different values is preferable to the current world state (e.g. if the world is clearly heading towards self-destruction which would terrify even the god).
Sure. Maybe you misunderstood me. I didn’t write this as an endorsement of the “even deeper atheism” view.
First of all, there might exist cases when a god with different values is preferable to the current world state (e.g. if the world is clearly heading towards self-destruction which would terrify even the god).
It still seems like this is much more prevalent in ratfic. E.g., compare how much of it you see in the top 10 ratfic works vs IDK top 10 generic high fantasy or top 10 post-human scifi works.
I have unvoted this comment because I can’t decide whether I feel happy that I posted it. However, I did feel it was important to leave it here anyway.
I agree with the complaint about rationalist fiction. Your choice of concern example is understandable and I would also find it disturbing if I experienced it. I have a similar sense of disturbing feeling when considering the memetics of other modern ideologies, and I hope to someday become confident that your choice of who to criticize does not have a systematic exception. I can’t tell if it’s real, but I have a sense of isolated demand for rigor when you pick on the left and center but not the right.
it seems to me that left vs right isn’t a particularly important dimension compared to the dimensions of auth-vs-liberty, prosocial-vs-antisocial, and egalitarian-vs-takeovertheworldism, that we should be focusing on broad-spectrum anti-authoritarianism and prevention of power concentration; in which case, I would hope you can also criticize authoritarianism on the right. But what I see is someone who endorses anti-egalitarianism and hasn’t visibly engaged with the value prop for egalitarianism or how you would achieve value satisfaction for the motivations for it in a broad-spectrum, cross-view-compatible way. If I felt my views were welcome in a coalition that included you, I would be quite excited; it seems to me that you have the seed of something that could become a real alternative to the locked-in frameworks that are common today. But I see you prematurely associating it with a particular aesthetic in a way that concerns me, such that every time you post something, it seems to contain a sharp jab against the left without any matching pattern of sharp jabs against the right, whereas I see both as similarly broken in opposing parts of their worldviews: the left perhaps might be broken about how to make good things happen, the right might be broken about what good things are, for example. I do not endorse that claim fully because there are also brokennesses about what good things are on the left, and brokenesses about how to achieve good things on the right.
Alternate explanation: ratfic tends to come out this way because it prioritizes a certain kind of economic / game-theoretic realism, and in the long term multi-polar equilibria just aren’t that realistic in many settings (including, IMO, our present non-fictional setting...).
Like, regardless of how good / bad / ethical taking absolute power is, it’s often inevitable that some entity or faction will end up winning decisively, or at least negotiating some kind of lasting truce or grand bargain, where the stakes, outcome, and enforcement are determined by hard power.
Another subversion example:
Three Worlds Collide comes to mind as another partial subversion of the trope, and also illustration of my point—the superhappies are the ones trying to impose absolute power on the other civilizations in the story, and the humans in the true ending blow up their own planet just to be left alone. But once you’re in a setting with intergalactic civilizations, conflicting values, and access to WMDs, there’s no way to avoid reckoning with hard power and decisive outcomes.
Alternate explanation: ratfic tends to come out this way because it prioritizes a certain kind of economic / game-theoretic realism, and in the long term multi-polar equilibria just aren’t that realistic in many settings (including, IMO, our present non-fictional setting...).
I generally think rationalists gesture at this as inevitable more than they actually demonstrate that it is inevitable; i.e., long term multi-polar equilibria has been quite sticky for Westphalian states or for plankton.
Eh, conversely I think that historical examples that are not at the technological and competitive frontier are not very useful for reasoning about the limiting behavior and outcomes of AGI. History, nature, and business are full of examples of both unipolar and multipolar dynamics that were at equilibria for a while… until they were disrupted in some form or another, often forcefully and suddenly.
Another strain of thought from the early days of OB/LW is that the only or main alternative to a Singleton in the long run are Malthusian scenarios. I remember writing Non-Malthusian Scenarios (2009) to push back against this, but looking at it now, most of the non-Malthusian/non-Singleton scenarios aren’t actually that plausible or attractive.
> That is, in the worst case they could just behave exactly like a pure replicator. And they could do this without actually surrendering their values. So any argument of the form “there is no way anything that cares about us can survive in Malthusian equilibrium” seems false.
I think it’s quite plausible that this is actually not possible, i.e., either at technological maturity or in the runup to it, transmitting values like caring about humans into the next generation of agents is actually difficult or costly enough that such agents are outcompeted and disappear.
Another concern I have about Malthusian scenarios (beyond “deadweight loss” in your post) is that there will be an astronomical number of agents (potential moral patients) with little surplus to spend on things aside from survival and reproduction. What if they have net negative lives, and either negative utilitarianism is true, or there isn’t enough overall surplus to make the universe net positive?
I noticed this a long time ago and tried to write a ratfic that didn’t have this dynamic; I didn’t get particularly far, mostly because I don’t think I’m that great of a fiction writer.
I think a large part of this is which settings ratfic writers choose to write fanfic in. It is very easy to take JK Rowling’s Harry Potter setting and put ratfic in it, because it’s about as screwed up as the real world, and that calls for making major changes / doesn’t naturally call for fitting into the system.
I was writing a MLP fanfic, and the My Little Pony setting is way less screwed up; the protagonist, rather than being a frustrated genius who isn’t taken seriously by his parents or teachers, is a pampered prodigy who great things are expected of and whose education is being carefully attended to accordingly. If Twilight goes to Celestia with some complaint about how society is arranged, Celestia encourages her to write a memo to the relevant minister and then get into a policy debate which considers all of the relevant factors.
[Separately I tried writing Warhammer ratfic, which mostly turned into a meditation on how much it sucks to be in an epistemically hostile environment, and the Empire was already doing a mostly-optimal strategy given the existence of the ruinous powers. But that’s, like, a short story’s worth of content.]
I think another part of it is… lack of comfort with responsibility? In the narrow, local sense which I think makes for a good minister or romantic partner but is not the heroic responsibility of the CEO or God-Emperor or whatever.
I think another part of it is… lack of comfort with responsibility? In the narrow, local sense which I think makes for a good minister or romantic partner but is not the heroic responsibility of the CEO or God-Emperor or whatever.
Yeah I’m probably trying to pack too many things in together. To expand on it:
I think there’s something that one can get from, for example, taking care of a garden, or a tank of shrimp, or whatever. Rationality helps a lot with it; you need to notice things, you often need to sweep away your preconceptions, you often need to rearrange how you orient to the world.
Harry really doesn’t demonstrate much of that; he couldn’t
keep alive a pet rock, after all.
And I think as you go thru the list of ratfic heroes, most of them also don’t have these sorts of responsibilities, or have them in a way that advances the plot instead of being the plot. (Miles Vorkosigan makes a lot of his feudal duties, but I think would very much not seem like a hero to a feudal audience, instead of something more like a tribal trickster deity.)
Part of this also is that it’s an ongoing relationship. You don’t get your pet to a good state and then declare mission accomplished; you instead have it occupying a bit of your attention, adjusting it as necessary. There’s a way it’s larger than what fits into your models in a way that is often breaking and expanding them, rather than being something that you can fit into your models and brilliant path your way around. (If Ender can, thru flexibility of mind, defeat the battle school, this is, in some sense, evidence that battle school was not a strong enough enemy for Ender.)
[Maybe another take on this is: ‘something to protect’ as the plot instead of the character’s motivation for getting good at punching is a pretty different type of story!]
Ratfic typically thinks of improving the world as a selection problem. Selecting a better world from the space of possible worlds is neat and elegant and lets you solve all problems at once. The only issue is that you need to gain absolute power first in order to be able to select the future you want.
Whereas you can also think about improving the world as a control problem, where you’re gradually nudging the world towards being better. This is less narratively satisfying when you’re a highly systematizing thinker, because you want to be able to identify the single root problem and take it out in one fell swoop (the same style of thinking that the communists were doing). But it’s much more robust when you’re in a world full of other people all of whom are also trying to exert influence.
IIRC none of Harry in HPMoR, Aaron in Unsong or Naruto in Waves Arisen actually meaningfully improved the world before taking it over—if anything, they mostly made it worse.
(Harder to evaluate this for the r!Animorphs or Keltham, because they were operating in such adversarial environments. And I don’t remember Worth the Candle well enough to say one way or the other.)
I agree this is overall kinda concerning and suggestive of a blindspot. But, I want to somewhat disagree about Planecrash.
The thing that happens there feels like “A small, powerless country goes about inventing nukes, demands it gets invited to the UN-equivalent of people who already have nukes, and, then negotiates with the rest of the people there on even-ish terms. It was already the case that the UN was dominated by people with enough military power to fuck shit up and negotiations therebetween.”
and, I certainly see the “rationalist wants to take over and make everything clean” aesthetic there, but it at least didn’t feel like it was failing to model the negotiation process at all.
Your analogy seems a bit skewed because on the scale “how much of a world takeover is this?”, “gaining the ability to unilaterally destroy the world” scores much higher than merely “gaining nukes”. If becoming a nuclear power let you unilaterally destroy the world, the US would have tried much harder to limit their spread!
It seems more like “a small powerless country seizes the USSR’s entire nuclear weapons stockpile (or creates an equivalently large one of their own) and tells everyone that they’ll cause nuclear armageddon unless their demands are satisfied”. Which is pretty world-takeover-adjacent even if it’s not exactly “taking over the world” in the classic sense.
(I’d also describe it as a central example of “gaining the power to design a new world order”, but not a central example of “gaining the power to design a new world order from scratch”.)
I think the existence of Hell is incredibly morally relevant and also very game-theoretically relevant too. I probably don’t need to elaborate on the morality side, but for the game theory: IIUC, Keltham really would prefer the universe be destroyed (with him in it!) than that Hell continue to exist. This is different from the stereotypical doomsday threat, which is made by someone who doesn’t actually want doomsday to happen but is hoping that other people will fear it even more and cave.
Yes, I agree. However, as I mentioned in my OP, I think that the prominence of Hell in stories like Unsong and Project Lawful is partly due to them functioning as plot devices to make taking over the world not just ethical but in fact morally obligatory.
Analogously, if a bunch of 19th-century Marxist fiction featured working conditions far harsher than any that existed in the real world, which compelled the heroes to launch a proletarian revolution, you wouldn’t just think “this makes total sense given the fictional premise”, you’d also think “the fictional premise was chosen to help the authors make the thing they already supported (and wanted to write about) seem morally good”.
And “take over the world for good reasons” was IIRC MIRI’s actual plan (hidden under the terminology “decisive strategic advantage” or “pivotal act” or similar).
Ok, but factory farms really do exist? And I don’t think it’s an exaggeration to call them hellish, or hells-on-earth.
And most of the beings that live in our civilization, depending on how you count, live and die in those factory farms.
So it seems that our world is not very different from that of Unsong or Project Lawful, in this respect? Even if the authors chose that setting for literary convenience, and it is only accidentally reflective of reality.
I’d say ratfics are more about becoming God, and as God you can naturally Fix the world. So you can view rats as atheists who believe that since God doesn’t exist, we must build Him.
Edit: Really, ratfics are about becoming more you are, with becoming God as the natural limit.
This phrase reminds me of a Russian sci-fi piece literally named Hard to Be a God. I expect this piece to be relevant, but I find it hard to explain the relevance without spoilers.
[Haven’t read those fictions or the quoted spoilers]
Glad to hear you saying this. In fact, I’ve had a couple quite concerning conversations with a couple different people in which I’m like “so what would you do if you accidentally invented a Friendly AGI in your basement one day? what are some of your important first actions?” and they don’t give one of the incredibly obvious and important answers, and don’t even necessarily agree to it after I say it.
I’ll add that the HM attitude extends to various other things. For example, some people around here are contemptuous of bioethics. I can see where they’re coming from, but I think the attitude is quite wrong, and in particular, one does want to “bring in many stakeholders to the conversation as coequal voices”, because that’s how you be non-HM. Cf. https://www.lesswrong.com/posts/yH9FtLgPJxbimamKg/genomic-emancipation-contra-eugenics
one does want to “bring in many stakeholders to the conversation as coequal voices”, because that’s how you be non-HM
I think it’s quite shameful how rationalists aim to optimize the world and yet are generally uninterested in what life is like for many kinds of normal people, or what problems they face. To rule well, you should know what life is like for your subjects, and what they care about, at minimum.
But there’s also the pattern where “if your users tell you about a problem, they’re always right, and if they tell you the solution, they’re almost always wrong.” And many common forms of “consulting all the stakeholders” ends up giving a veto to special interests, which ends up strangling liberty.
There are some to my knowledge unsolved problems here.
To rule well, you should know what life is like for your subjects, and what they care about, at minimum.
This is still totally High Modernist. To actually rule well you should actually not try to rule as much and should actually try to share power, which includes giving up power. Not giving up all power, but, you know, you leave after your second term in office, and you create a parliament.
For example, some people around here are contemptuous of bioethics.
I do wonder how much of this is “contemptuous of bioethics done badly”. I can see the argument for “that means we need to do bioethics well, not continue to cede ground” but I do think it’s important to be honest about when fields are failing.
it’s important to be honest about when fields are failing.
I agree, naturally. I criticize bioethicists as a group, precisely because they are, for the most part, AFAICT, failing to lead on moral questions around reprogenetics.
However, I think one has to make multiple updates. That observation indicates that bioethicists may not have the right abilities or motivations or other properties (determination, grit, courage, sanity, wisdom, what have you), and on those grounds could be dismissed as individuals or even as a group; but it also indicates that the problems themselves are especially difficult. The latter is often underappreciated. To avoid “HM bioethics” you actually have to cede power in the discussions, and even logistically doing that is difficult (I mean, I don’t know how to do it; I don’t have a near-fully-satisfying theory of how to give proper / coequal decision weight to all the stakeholders who should have that).
I think part of the HM mindset is precisely reacting to “a bunch of other people are doing it bad and kinda punishing me for trying to do it better” with “actually I should just be in charge and not worry about the concerns those people talk about” rather than “some other group of people would have to figure out how to do it better”.
I was going to say that this seems like another one of Eliezer’s founder effects, but he actually wrote about not trusting humans with too much power, in Creating Friendly AI 1.0 (2001):
Among humans, the only practical way to maximize actual freedom (the percentage of actions executed without interference) is to ensure that no human entity has the ability to interfere with you—a consequence of humans having an innate, evolved tendency to abuse power. Thus, a lot of our ethical guidelines (especially the ones we’ve come up with in the twentieth century) state that it’s wrong to acquire too much power.
If this is one of those things that simply doesn’t apply in the spaces beyond the Singularity—if, having no evolved tendency to abuse power, no injunction against the accumulation of power is necessary—one of the possible resolutions of the Singularity would be the Sysop Scenario. The initial seed-AI-turned-Friendly-superintelligence, the Transition Guide, would create (or self-modify into) a superintelligence that would act as the underlying operating system for all the matter in human space—a Sysop.
I wonder how to interpret e.g. the ending of HPMOR in light of this.
HJPEV is bound by a magical oath that prevents this human failing in the same way it is prevented in an agent that meets tiling desiderata. This is explicit in the text. E-Book draft, 2015, chapter 113.
Admittedly this both assumes that the “time of peril” hypothesis is correct and can be handled while maintaing human freedom, and the solution only (in maximum robustness) binds until the end of this time.
My understanding of HPMOR is limited as I’ve only read a few chapters, but looking up the text you cite, it doesn’t seem to prevent most forms of abuse of power.
On the other hand, 3 Worlds Collide is an interesting case study:
The first ending is essentially a “use godlike tech-powers to optimize the world” scenario, except that it’s being carried out by the superhappies, rather than a human protagonist. The superhappies do actually care about human wellbeing and try to compromise with human values to the extent they feel possible. Then the second ending basically says, “yeah, regardless of the intended-benevolence of the superhappies, it’s worth blowing up an inhabited star system to avoid being ruled by that kind of god”.
Your overall point seems true and important. The virtues that are enshrined by the rat-fic tradition are primarily “sanity in the midst of insanity (eg. social decoupling and thinking for yourself)” and “heroic responsibility”. These attitudes are not strictly counter to cooperation or working with the existing systems of the world, but they sure do tend to push in the opposite direction. The more you distrust the world, the less you think that you should cooperate with it.
And further, actual practice, “thinking for yourself” directly cuts against human political coordination, since political coalitions almost always maintain unity by coordinating their beliefs, and challenging those coordination beliefs undermines the coalition.
(I find it amusing when people sometimes say that the “one job” of rationalists is to be able to coordinate. I think that’s ass-backwards. The rationalist tradition is about prioritizing independence of thought and the epistemology to discern the truth at the expense of human coordination. There’s some hope there’s another even stronger mode of coordination on the other side of the uncanny valley, but that, so far, remains an unvalidated hope.[1])
However, with regards to this specific point...
But the worlds didn’t have to be that bad! The writers chose to design them that way—I expect in significant part because that provides a narratively compelling backdrop for the thing they wanted to write about, which was their heroes taking over the world.
Eliezer has said that one of the reasons he writes fanfiction, is that he doesn’t have to invent the world. All of the horror and badness was already present in the source material.
Azkaban, as Rowling depicted it in the wizard world, is entirely realistic. If there are no Dementors in American prisons, it’s because American politicians have no Dementors to use, not because they’re better people than the Wizengamot. Sexual assault is routine in American prisons and that could easily be prevented with video cameras. American prisons are worse than Azkaban in ways that Rowling couldn’t easily have imitated without breaking her readers’ suspension of disbelief. At least the wizarding world isn’t imprisoning marijuana users that we ever saw.
Even so, if Azkaban were in a world of my own invention, someone might question the realism of Harry’s reaction to Azkaban, versus other people in magical Britain seeming not to notice Azkaban as a moral horror. (Just like Americans don’t notice the moral horror! Rowling was not being unrealistic!) How is it that Harry sees all these utilons that can be picked up by ending Azkaban, that nobody else has seen? (Answer: it is not possible for any arbitrary economic actor to make a hundred thousand Galleons of profit if they have the insight that Azkaban is needlessly cruel, so standard economics does not predict moral efficiency the way it predicts efficient stock markets.) Perhaps Eliezer Yudkowsky only invented Azkaban to be triumphed over by his allegedly superior hero, and put it into his world as a straw inefficiency, a weakman…
But I didn’t invent Azkaban, it’s right there in canon, and millions of readers read J. K. Rowling’s stories and (correctly) accepted this as a routine background premise rather than claiming (incorrectly) that no (flawed) democracy (the size of a small town) would ever do such a thing and that she was just putting Azkaban there to show off her hero’s moral superiority.
Indeed, a hope that I personally share. But think the folks who are inclined to say that coordination is the “one job”, or similar, of rationalists are missing the plot. Coordinating groups larger than 50 people to accomplish a goal or a political change, by way of accurate views instead of adaptive stories and taboo beliefs, is an unsolved challenge.
I’ll note that overall, the EAs seem to be doing somewhat better than the rationalists at working in groups to get stuff done, and also, they are relatively less free-thinking, and have more taboo beliefs.
Eliezer has said that one of the reasons he writes fanfiction, is that he doesn’t have to invent the world. All of the horror and badness was already present in the source material.
A large majority of fantasy settings don’t have literal hells as a key component, so I think my point is still applicable to Project Lawful if you replace “design them that way” with the more general “select for that trait”.
I do agree that this is a good point with regards to HPMoR, which is one reason why I didn’t include HPMoR in my original list of examples.
Aaron doesn’t take over the world alone. He merges with seven other wildly different minds, including the villainous Dylan Alvarez. “In William Blake’s prophecies, Albion was the entity formed at the end of time, when all of the different aspects of the human soul finally came together to remake the world”, as one of them says.
And I don’t think the ending is about recreating the world as some kind of rationalist utopia (how would you do that with Dylan an Erica on the team?) - I interpret it more as a “cycle continues” ending where they carry forward God’s already perfect plan into a new world.
See for example this point in the Tosefta, where Scott explains all the Easter eggs:
“As for THARMAS, seven of the ten towers were smoking ruins; the other three were heavily scarred. In the epilogue, THARMAS is going to be used to make the new universe. Seven of ten towers destroyed plus the rest damaged = seven of ten sephirot cracked plus the rest damaged, indicating the new universe will work the same as our own.”
I basically agree with what you notice, and think that this is what you’d expect if rationalists were mostly normal relative to other people in their goals, which are mostly selfish and dicatorial, but are more intelligent and can think farther ahead about what instrumental goals that they imply for their terminal goals.
Or put another way, the thing that rationalists are doing here are things lots of other people would likely do if they were more intelligent, and the truth of the matter is that most people just like all-powerful dictatorships, almost no matter their ideology.
This is pretty straightforwardly not true, there are plenty of academics (for example) who are as smart as rationalists but don’t do very broad instrumental reasoning.
There are also plenty of people who don’t fantasize about becoming all-powerful dictators.
I think that the hunger to become god is an unusually rationalist trait. Honestly it’s somewhat reminiscent of sociopathy, but fortunately few rationalists seem to be sociopaths. However, I do think a sufficient level of fear of death causes some overlapping traits, e.g. a mentality in which more power is crucial to solving problems. (This is not meant in a particularly blame-y way, I’m just as much an example of this as anyone else around here.)
This is pretty straightforwardly not true, there are plenty of academics (for example) who are as smart as rationalists but don’t do very broad instrumental reasoning.
Fair point, I was generalizing too much here.
I agree with the literal claim that plenty of people don’t fantasize about becoming all-powerful dictators, but I’d say the percentage of people who don’t fantasize (including in their heads and not speaking about it) becoming dictators or don’t believe that an all-powerful dictator is necessary to solve problems/have a good future is much closer to 25-30% than 90% or more here, and this is more of an upper bound than a lower bound.
The reasons for why this is the case partially delve into politics that would cause way more heat than light if I discussed it on here, but one of the reasons for this is that for a lot of citizens, they don’t want to get involved in politics and want someone else to solve their problems for them, and one of the unique traits of a lot of non-dictatorial systems of government is that the average person has to be more involved with politics, and lots of people hate doing this.
An all-powerful dictator where average citizens make none of the decisions in a new world order doesn’t require them to pay attention to their government/politicians, and a lot of people genuinely want the ability to not care about politics at all.
There’s a strong pattern in ratfic of the protagonist “winning” by gaining the power to design a new world order from scratch—i.e. taking over the world. It’s a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it’s hard to unsee.
You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn’t have to be that bad! The writers chose to design them that way—I expect in significant part because that provides a narratively compelling backdrop for the thing they wanted to write about, which was their heroes taking over the world.
From my current perspective, this feels ominous in a kinda similar way as if I read a bunch of Marxist fiction from the 1800s and noticed that it always ended in a communist revolution.
Spoilers for a range of ratfic (unfortunately telling you which ratfic itself constitutes a spoiler):
EDIT: the spoiler tags seem not to be working on the images. Mods, can you help? For now I’ll add a bunch of space below. Expect many spoilers if you scroll down.
Unsong:
Worth the Candle:
The Waves Arisen:
r!Animorphs: The Reckoning
Project Lawful:
And some examples which (at least partially) subvert this trope:
Friendship is Optimal and Branches on the Tree of Time both involve world takeovers with some tension about how good or bad they are. In the former case, that tension is the main thrust of the story. In the latter case, it’s a bit more ambiguous: the ending sure seems pretty ominous, but it’s also exactly what the heroes were aiming for, and I’m not noticing much authorial prompting to question their reasoning.
The Metamorphosis of Prime Intellect is much more explicit about the “taking over the world” thing being bad. But it also came out in the 90s. Interesting that (to my knowledge) no rationalist has written a story which is so straightforward about the downsides of a well-intentioned world takeover.
My own contribution is equally telling: https://tomasbjartur.bearblog.dev/rational-teletubbies/
I also liked this one for foreshadowing the same problems Ngo was complaining about: https://substack.com/home/post/p-179661551
lol, incredible.
Perhaps the direction of causation goes the other way? In the real world, it seems like we are headed towards a situation where 1-3 giant armies of superintelligent AIs transform the planet. Whoever controls those AIs, if anyone does, will be in a position analogous to all those fictional examples you give.
All the authors of all those works of fiction know this, or at least have been seriously entertaining this AI-power-concentration-hypothesis for years.
So, they are writing fictional analogies for the situation they expect to actually happen in real life. Except of course, since they are writing fiction, it has to have a happy ending.
Their expectation that it’s going to happen in real life is in fact correct unfortunately, as best as I can tell. Will the ending in real life be as happy as the endings in fiction? Of course not, but they shouldn’t be taken as implicitly claiming that, anymore than Tolkien should be taken as implicitly claiming that all this mechanization, industrialization, technology, etc. that he was reacting to was going to end happily ever after in real life thanks to the plucky heroism of some country bumpkins who end up destroying it all.
Well, exactly what I’m disputing here is how happy the ending is. For example, imagine that all of these stories played out exactly the same, with the exact same amount of concentration of power. But instead of the heroes getting to use that power to reshape the world, the power instead goes to.… a random person off the street. I expect that these authors, if they were to write that kind of ending, would portray it as a maybe-happy-ish ending, but one that’s still pretty scary and uncertain.
And indeed, this is roughly how I’d describe the stories mentioned above where a mostly-aligned AI gets total power—Friendship is Optimal, Branches on the Tree of Time, and Metamorphosis of Prime Intellect. These stories really grapple with the sense of unease and tension that comes with almost everyone losing almost all their power.
Whereas when I look at the examples of ratfic above, the stance they’re taking seems to be “our heroes became dictators of the universe. This is a straightforwardly happy ending.” And indeed, on several occasions (maybe as many as half a dozen?) I’ve heard people describe the ending of Worth the Candle as one of the best utopias they can imagine. All of this really seems like a big ideological blind spot.
I don’t think I understand how you are objecting to my point. Straightforwardly happy endings are the norm in fiction. I think you are reading too much into their choice to make the ending straightforwardly happy.
Let’s think about, idk, the Star Wars series for comparison. In it, the plucky heroes go to war against the bad guys, kill loads of them in a series of military engagements, and ultimately emerge victorious with their faction the new rulers of the galaxy. This is presented as straightforwardly happy ending.
In fact lots of mainstream popular stories are of this pattern: The heroes defeat the bad guys in armed combat, kill loads of them, and then their faction has more political and military power afterwards and they all live happily and it’s great and uncomplicated.
Applying your reasoning to these popular stories, I guess you’d say: “Ominous. It seems like most Hollywood writers expect there to be a giant war in the future, possibly a civil war or series of assassinations. They are encouraging people to fight in this war / battle / revolution / assassination-series, and win, and they are portraying this as uncomplicatedly good.”
But I feel like there’s a better explanation for what’s going on: (a) Their audience likes fighting as a way of resolving conflicts, it’s consistently entertaining and interesting to most people, so that’s why so many movies conflicts resolve via combat, and (b) their endings are uncomplicatedly happy endings because again that’s what most audiences want, that’s the standard way to make a satisfying story.
I do actually think that the general trope of “the rebels winning is sufficient for a happy ending” is pretty indicative of poor ethical thinking.
But even Hollywood balks at their heroes ending up with literal godlike control of the world. For example (though I haven’t watched the series) my impression of the Avengers franchise is that they introduce a plot device (the infinity gauntlet) that gives its wielder godlike powers, the heroes use it specifically to defeat the bad guy and undo the damage he caused, and then they destroy the device.
In other words, they got to the exact point that ratfic heroes got to, and then their happy ending specifically involves them giving up the same kind of godlike power that ratfic heroes typically use to make themselves dictators of the universe.
Similarly for Superman: his happy endings involve him successfully using his godlike powers to beat the bad guys without changing the established world power structures basically at all. And I feel pretty confident that a big reason Superman doesn’t end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending.
tl;dr: there are many ways to make a story have a happy ending, and it’s quite indicative of the authors’ ethical and political views which endings they consider to be happy. The kind of endings that rationalists often portray as happy, mainstream scriptwriters seem to go out of their way to avoid.
Yes. And I claim they’re wrong about that.
There’s lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help.
If I ended up with “ultimate power” somehow, by some mechanism that didn’t involve me taking on ultimate power for a specific narrow mandate, I think it is both ethically correct to use it to permanently end many (but probably not all) of those evils.
This is indeed pretty scary.
Oh, I think of “ending factory farming” as very far from “taking over the world”.
If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.
Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren’t laws against, like flying around them in a way which blows the buildings over).
This might escalate to the government trying to arrest him, and outright conflict, but honestly if Superman isn’t skillful enough to defuse that kind of thing, given his influence, then he doesn’t have much business imposing political changes on the world anyway. A politically unskilled and/or unvirtuous Superman trying to end factory farming could quite easily destabilize society in a way that is far worse long-term than letting factory farming end on whatever the natural counterfactual timeline is (without AI, maybe 20 or 30 years?)
Relatedly I’m increasingly coming to believe that this reasoning applies to Lincoln, and that we’d be in a much better position if he’d let the Confederacy secede and then imposed strong economic and moral pressure on them to end slavery.
Why? Because that was the beginning of the centralization of power with the federal government?
I think Alexander Hamilton was the beginning, but this seems like a big step. Vassar talks about how, from the civil war onwards, the American legal system needed to be optimized to rule a vassal state while also pretending that they weren’t ruling a vassal state. Can’t remember the specific examples he cited to me but I found it fairly compelling.
This certainly seems to be the case with Trump’s (in my understanding) limited ability to govern blue states and cities.
This seems like a weird set of claims to me.
First of all, yes, it seems very likely to escalate to conflict? Many nations would consider Superman’s property damage to their factories to be an act of war by a foreign power, and that he’s doing it in ways that technically circumvent existing laws is not going to make much of a difference. (That seems more annoying actually.)
He could try to negotiate for big, universal, changes to the food supply chain via diplomatic channels, backed by offers to trade various things he can provide, which generally seems like a better way to go about it. But if push comes to shove, I think he should back up his demands with effectively military intervention, if that could work. (Actually, my guess is that if he’s resorting to violence, he should mainly be relying on targeted assassinations.)
Also, the factory farming is not the only evil that our hypothetical superman should end. I think he should probably initiate regime changes in various authoritarian or dictatorial countries (eg North Korea, Eritrea, Afghanistan), or otherwise enable residents to leave the territory of those regimes. Again, he should attempt to negotiate for that diplomatically, first, but this is very clearly against the core interests of those regimes, and it seems very likely that he’s going to end up going to war with them.
Are you saying that if the diplomatic negotiations deteriorate to the point of military action, that means that our hypothetical superman has failed, and he would be better off retiring? Don’t existing legitimate countries go to war for far less noble reasons all the time?
I do think that having a superman around is extremely scary, and among other things, he should make the standards that he’s enforcing extremely clear and legible. He should make a point to never be capricious. He should write clear manifestos declaring the moral principles that he’s upholding and the logic behind them. And if his principles change for some reason, he should telegraph that very clearly. He should telegraph his actions with a lot of lead time, so that he’s a predictable agent to make plans around.
But ultimately he should declare:
This is basically how I would want a hegemonic morally-motivated nation-state to behave.
What standard do you want to hold our superman / morally-motivated nation state to?
I indended this to refer to scenarios where the US itself (or other leading western powers) were taking military action against Superman. I care much less about whether he destabilizes North Korea or Eritrea or even countries similar to those but better-governed. But I care a lot about whether he destabilizes the countries I consider the best and most important ones.
Maybe. Or maybe they really wouldn’t want to pick a fight with Superman. Or maybe they would issue an angry press release then not do anything. In a setting where Superman holds basically all the cards in terms of physical force, most nations would try quite hard to defuse tensions with him (unless, as I discussed, he’s very unskilled).
I’m still not sure what standard you’re holding our hypothetical superman too. You just don’t want him to destabilize the countries that you consider most important?
If he overthrows the Communist Party of China, is he violating your standards?
If he forcibly institutes electoral reforms to make the US government more functional, is that violating your standards? Is the “forcibly” there a problem, and if it comes to using military force to push for electoral reform, he should hold off?
What’s the line that he shouldn’t cross?
I claim that there are some people such that, if they were dictators of China, that would be much worse than the current situation. And there are some people such that, if they were dictators of China, that would be much better than the current situation. Which category a given person falls into depends a lot on their honesty, integrity, wisdom, ability to understand political dynamics, ability to resist manipulation, etc.
There are no particular limits I’d want to place on a sufficiently virtuous Superman. E.g. I want Superman to follow a policy that leads him to overthrow the government of China iff he is in the latter category. The big question is how Superman can gain justified confidence that he’s in the latter category, given that unvirtuous people are prone to a lot of self-deception. One way he can do it is by setting limits on his own behavior so that he can gain more evidence about what kind of person he is. E.g. maybe he thinks he’s really wise about politics—wise enough that him having control over US electoral policy is a good idea. If so, he should try to test that wisdom by implementing political change without using violence. If he starts telling you that he doesn’t need to pass such tests, because he’s already so confident that his plan is a good idea, then you should start getting worried.
In other words, when I think about a question like “should Superman forcibly institutes electoral reforms to make the US government more functional”, I expect that there are some ways to do this that are really good, and some ways to do this that are really bad. And the kinds of people who are capable of doing it in a really good way (given that they’re Superman) are also generally the kinds of people who wouldn’t need to use much force to make it happen (given that they’re Superman).
Ok, I agree with all this!
(Modulo, I have more uncertainty about how much force the wisest path entails when, one has hegemonic power. Certainly using military force to get your way has major costs, and so, taking those costs into account, I would expect the wiser courses of actions to be more peaceful, generally.)
Should I then summarize your criticism of ratfic protagonists as something like:
Yes, great summary, I fully endorse it.
I’m also reminded of something that Val used to say: “power felt is power wasted”. Deploying a lot of force to make things go your way is very inefficient.
If you’re skilled, you should be able to get what you’re aiming for while deploying very little actual force (“speak softly and carry a big stick” for instance, but also using soft power and good leadership more generally). Someone with a little power and a lot of skill can often do as much (and with less collateral damage) as someone with a lot of power trying to muscle through.
Power is definitely useful, but the more you think that the thing you need to accomplish your aims, the more that indicates that you don’t have skill with using power efficiently.
I broadly agree with this comment too, though not as much as I agree with the other one.
Power felt can also be a kind of honesty—e.g. if a law is backed by force, then it’s often better for this to be unambiguous, so that people can track the actual landscape of power.
(Of course, being unambiguous about how much force backs up your laws can also be a kind of power move. I expect that there are ways to get the benefits of honesty without making it a power move, but I don’t have enough experience with this to be confident.)
In other words, I expect that the kind of inefficiency Val is talking about here is actually sometimes load-bearing for accountability.
This is very optimistic. Why do you think so? Alt. proteins outcompeting factory farmed meat?
He might be picking it from Jeff Sebo et al. https://verfassungsblog.de/global-ban-on-industrial-animal-agriculture/
Can you elaborate on that? I’ve never heard anyone say that before.
This version of the criticism I do kinda agree with. However I think it’s relatively mild.
Brandon Sanderson’s books have some interesting variants of this as well.
Spoilers for the Mistborn series (to the best of my recollection and with some consulting of the fandom wiki) (putting in collapsible for now, because I can’t get spoiler blocks to work with this new editor):
At the end of The Well of Ascension, Vin gets a hold of the power of Ruin (one of the Shards, roughly god-like entities of the universe). She knows that she can use it to eliminate lots of the world’s atrocities etc. Sazed, informed by a misty ghost, tells her that “it’s a trap” and she needs to release the power. Some other misty ghost appears out of nowhere and stabs Elend, nearly lethally, apparently to force Vin to accept the power, so that she can save her loved one. But she doesn’t do that, releasing the power, which turns out to free its prior weilder who is the bad guy and now goes on to destroy the world. He was also the spirit that told Sazed to stop Vin. Whereas the ghost that stabbed Elend was the Vessel of another shard, Preservation, a good guy (relative to Ruin, at least).
At the end of the next book, The Hero of Ages, Sazed accepts the powers of Ruin and Preservation, ascending to godhood and becoming Harmony. He approximately fixes the world, but then in the following series, it turns out that he’s not as powerful, as we might have predicted, and also the divine power that he has been wielding starts shaping his mind, so that he becomes more interested in things being Harmonious, than in “goodness”.
ETA: it’s plausibly relevant that Sazed, at the moment of Ascension, absorbed all the knowledge that had been stored for millennia in his Coppermind “amulets”, so that he could learn from the mistakes of the past generations and not screw things up as much.
IDK what the morale is supposed to be here, if any. “Sometimes you need to power-grab because otherwise someone else will power-grab in your place, but also beware because power corrupts and divine power corrupts in an ungodly way, so attaining instrumentally convergent goals is of limited value if it meddles with your utility function in unendorsed ways, be it due to some contingent peculiarity of your mental structure, or some more general fact of how minds work.”?
(There might also be something along those lines in other Sanderson cosmere books, but it’s been long since I’ve read any and I’m not up to date.)
ETA: Another relatedly interesting plot twist in Mistborn is that it turns out that the original bad guy, the Last Emperor, who gets killed at the end of the first book, imposed an oppressive regime in order to prevent humanity from Ruin. He also changed the planet for altruistic reasons that turned out to be misguided and hence harmful.
Very interesting.
Relatedly, A Practical Guide to Evil is one of my favorite books/series, and grapples with the tension between trust and power very well. It’s one of the very few narratives I’ve seen written skillfully enough that the protagonist giving up power didn’t seem straightforwardly stupid to me (even when I was in a classic rationalist mindset).
Interesting. Looks like plausibly in a similar vibe/direction as Venkatesh Rao’s Be Slightly Evil.
It seems plausible that what you suggest is one significant contributor. Here’s one more thing that imo plausibly contributes significantly:
Most of these people are consequentialists, i.e. they think of ethics in terms of sth like designing a good spacetime block. [1] Like, when making a decision, you are making a decision as if standing outside the universe and choosing which of two spacetime blocks [2] is better. Given this view of ethics, it is very natural to imagine a future in which there actually is some guy that designs/chooses a good spacetime block, and it becomes somewhat less natural to imagine futures in which the spacetime block keeps getting “designed/chosen” in a messy way by all the messy stuff inside the spacetime block, with the designing/choosing and the being-valuable done by the same entities. A person who thinks in terms of duties or a person who thinks in terms of virtues would find it much less natural to have such a strong separation between the locus of moral-agent-hood and the locus of moral-patient-hood.
or at least they are much more in this direction than the median person, or even the median person at their iq
or more precisely: two distributions on spacetime blocks
“Unilateralist consequentialists think they’re just temporarily embarrassed omnipotentates.”
There’s a story that the reason why you have a lot of anime that goes in very weird sexual directions is that Japan is an incredibly sexually repressive society, so the Japanese channel their compressed libido towards art.
I don’t really believe this story, but I think the pattern it exemplifies is at play here.
Rats have very ambitious goals of fixing the world. They see that the world is in a pretty bad shape and demands fixing. But they can’t fix it. No one else can fix it either (nihil supernum, deep atheism), and even if they could, it would likely be bad, because their values are not yours (even deeper atheism). So you yearn for a world in which you smash those limitations. “Power” (or a specific kind of it) is the thing you need, and its value is not bounded, so the threshold of a superstimulus is largely your imagination, perhaps constrained by some ontological assumptions. That’s why this theme recurs through ratfic so much.
This seems somewhat obvious to me[1] and so I am somewhat surprised that people in the comments mostly seem to explain it in terms of game-theoretic (or other) realism or people having consequentialist-ish views, etc.
partly through introspection + the Copernican principle, FWIW
First of all, there might exist cases when a god with different values is preferable to the current world state (e.g. if the world is clearly heading towards self-destruction which would terrify even the god). Additionally, I doubt that the theme of gaining power and fixing the world recurs just through ratfic and not through fiction whose authors display some bias which I cannot describe more precisely than “finding it hard to restrain themselves.” Finally, Max Harms has been trying to construct an agent whose sole goal is being corrigible and even defined power for the agent to optimize in a way which I suspect to be transformable into the Natural Abstract Goodness.
Sure. Maybe you misunderstood me. I didn’t write this as an endorsement of the “even deeper atheism” view.
Sure, it’s not an exclusively ratfic issue. https://tvtropes.org/pmwiki/pmwiki.php/Main/GodhoodSeeker
It still seems like this is much more prevalent in ratfic. E.g., compare how much of it you see in the top 10 ratfic works vs IDK top 10 generic high fantasy or top 10 post-human scifi works.
I have unvoted this comment because I can’t decide whether I feel happy that I posted it. However, I did feel it was important to leave it here anyway.
I agree with the complaint about rationalist fiction. Your choice of concern example is understandable and I would also find it disturbing if I experienced it. I have a similar sense of disturbing feeling when considering the memetics of other modern ideologies, and I hope to someday become confident that your choice of who to criticize does not have a systematic exception. I can’t tell if it’s real, but I have a sense of isolated demand for rigor when you pick on the left and center but not the right.
it seems to me that left vs right isn’t a particularly important dimension compared to the dimensions of auth-vs-liberty, prosocial-vs-antisocial, and egalitarian-vs-takeovertheworldism, that we should be focusing on broad-spectrum anti-authoritarianism and prevention of power concentration; in which case, I would hope you can also criticize authoritarianism on the right. But what I see is someone who endorses anti-egalitarianism and hasn’t visibly engaged with the value prop for egalitarianism or how you would achieve value satisfaction for the motivations for it in a broad-spectrum, cross-view-compatible way. If I felt my views were welcome in a coalition that included you, I would be quite excited; it seems to me that you have the seed of something that could become a real alternative to the locked-in frameworks that are common today. But I see you prematurely associating it with a particular aesthetic in a way that concerns me, such that every time you post something, it seems to contain a sharp jab against the left without any matching pattern of sharp jabs against the right, whereas I see both as similarly broken in opposing parts of their worldviews: the left perhaps might be broken about how to make good things happen, the right might be broken about what good things are, for example. I do not endorse that claim fully because there are also brokennesses about what good things are on the left, and brokenesses about how to achieve good things on the right.
Alternate explanation: ratfic tends to come out this way because it prioritizes a certain kind of economic / game-theoretic realism, and in the long term multi-polar equilibria just aren’t that realistic in many settings (including, IMO, our present non-fictional setting...).
Like, regardless of how good / bad / ethical taking absolute power is, it’s often inevitable that some entity or faction will end up winning decisively, or at least negotiating some kind of lasting truce or grand bargain, where the stakes, outcome, and enforcement are determined by hard power.
Another subversion example:
Three Worlds Collide comes to mind as another partial subversion of the trope, and also illustration of my point—the superhappies are the ones trying to impose absolute power on the other civilizations in the story, and the humans in the true ending blow up their own planet just to be left alone. But once you’re in a setting with intergalactic civilizations, conflicting values, and access to WMDs, there’s no way to avoid reckoning with hard power and decisive outcomes.
I generally think rationalists gesture at this as inevitable more than they actually demonstrate that it is inevitable; i.e., long term multi-polar equilibria has been quite sticky for Westphalian states or for plankton.
Eh, conversely I think that historical examples that are not at the technological and competitive frontier are not very useful for reasoning about the limiting behavior and outcomes of AGI. History, nature, and business are full of examples of both unipolar and multipolar dynamics that were at equilibria for a while… until they were disrupted in some form or another, often forcefully and suddenly.
Another strain of thought from the early days of OB/LW is that the only or main alternative to a Singleton in the long run are Malthusian scenarios. I remember writing Non-Malthusian Scenarios (2009) to push back against this, but looking at it now, most of the non-Malthusian/non-Singleton scenarios aren’t actually that plausible or attractive.
I wrote a bit about Malthus maybe having less teeth than he seems here: https://tomasbjartur.bearblog.dev/contra-my-own-doomerism/
Not particularly well thought out and I may be rehashing things, but I haven’t seen it put this way before.
> That is, in the worst case they could just behave exactly like a pure replicator. And they could do this without actually surrendering their values. So any argument of the form “there is no way anything that cares about us can survive in Malthusian equilibrium” seems false.
I think it’s quite plausible that this is actually not possible, i.e., either at technological maturity or in the runup to it, transmitting values like caring about humans into the next generation of agents is actually difficult or costly enough that such agents are outcompeted and disappear.
Another concern I have about Malthusian scenarios (beyond “deadweight loss” in your post) is that there will be an astronomical number of agents (potential moral patients) with little surplus to spend on things aside from survival and reproduction. What if they have net negative lives, and either negative utilitarianism is true, or there isn’t enough overall surplus to make the universe net positive?
I noticed this a long time ago and tried to write a ratfic that didn’t have this dynamic; I didn’t get particularly far, mostly because I don’t think I’m that great of a fiction writer.
I think a large part of this is which settings ratfic writers choose to write fanfic in. It is very easy to take JK Rowling’s Harry Potter setting and put ratfic in it, because it’s about as screwed up as the real world, and that calls for making major changes / doesn’t naturally call for fitting into the system.
I was writing a MLP fanfic, and the My Little Pony setting is way less screwed up; the protagonist, rather than being a frustrated genius who isn’t taken seriously by his parents or teachers, is a pampered prodigy who great things are expected of and whose education is being carefully attended to accordingly. If Twilight goes to Celestia with some complaint about how society is arranged, Celestia encourages her to write a memo to the relevant minister and then get into a policy debate which considers all of the relevant factors.
[Separately I tried writing Warhammer ratfic, which mostly turned into a meditation on how much it sucks to be in an epistemically hostile environment, and the Empire was already doing a mostly-optimal strategy given the existence of the ruinous powers. But that’s, like, a short story’s worth of content.]
I think another part of it is… lack of comfort with responsibility? In the narrow, local sense which I think makes for a good minister or romantic partner but is not the heroic responsibility of the CEO or God-Emperor or whatever.
I don’t quite get this bit.
Yeah I’m probably trying to pack too many things in together. To expand on it:
I think there’s something that one can get from, for example, taking care of a garden, or a tank of shrimp, or whatever. Rationality helps a lot with it; you need to notice things, you often need to sweep away your preconceptions, you often need to rearrange how you orient to the world.
Harry really doesn’t demonstrate much of that; he couldn’t
keep alive a pet rock, after all.
And I think as you go thru the list of ratfic heroes, most of them also don’t have these sorts of responsibilities, or have them in a way that advances the plot instead of being the plot. (Miles Vorkosigan makes a lot of his feudal duties, but I think would very much not seem like a hero to a feudal audience, instead of something more like a tribal trickster deity.)
Part of this also is that it’s an ongoing relationship. You don’t get your pet to a good state and then declare mission accomplished; you instead have it occupying a bit of your attention, adjusting it as necessary. There’s a way it’s larger than what fits into your models in a way that is often breaking and expanding them, rather than being something that you can fit into your models and brilliant path your way around. (If Ender can, thru flexibility of mind, defeat the battle school, this is, in some sense, evidence that battle school was not a strong enough enemy for Ender.)
[Maybe another take on this is: ‘something to protect’ as the plot instead of the character’s motivation for getting good at punching is a pretty different type of story!]
Another way of putting this is in terms of the distinction between two types of optimization: selection and control.
Ratfic typically thinks of improving the world as a selection problem. Selecting a better world from the space of possible worlds is neat and elegant and lets you solve all problems at once. The only issue is that you need to gain absolute power first in order to be able to select the future you want.
Whereas you can also think about improving the world as a control problem, where you’re gradually nudging the world towards being better. This is less narratively satisfying when you’re a highly systematizing thinker, because you want to be able to identify the single root problem and take it out in one fell swoop (the same style of thinking that the communists were doing). But it’s much more robust when you’re in a world full of other people all of whom are also trying to exert influence.
IIRC none of Harry in HPMoR, Aaron in Unsong or Naruto in Waves Arisen actually meaningfully improved the world before taking it over—if anything, they mostly made it worse.
(Harder to evaluate this for the r!Animorphs or Keltham, because they were operating in such adversarial environments. And I don’t remember Worth the Candle well enough to say one way or the other.)
I think it’s… more like a wash? A lot of this depends on what you think about Anglecynn / its internal politics.
I agree this is overall kinda concerning and suggestive of a blindspot. But, I want to somewhat disagree about Planecrash.
The thing that happens there feels like “A small, powerless country goes about inventing nukes, demands it gets invited to the UN-equivalent of people who already have nukes, and, then negotiates with the rest of the people there on even-ish terms. It was already the case that the UN was dominated by people with enough military power to fuck shit up and negotiations therebetween.”
and, I certainly see the “rationalist wants to take over and make everything clean” aesthetic there, but it at least didn’t feel like it was failing to model the negotiation process at all.
Your analogy seems a bit skewed because on the scale “how much of a world takeover is this?”, “gaining the ability to unilaterally destroy the world” scores much higher than merely “gaining nukes”. If becoming a nuclear power let you unilaterally destroy the world, the US would have tried much harder to limit their spread!
It seems more like “a small powerless country seizes the USSR’s entire nuclear weapons stockpile (or creates an equivalently large one of their own) and tells everyone that they’ll cause nuclear armageddon unless their demands are satisfied”. Which is pretty world-takeover-adjacent even if it’s not exactly “taking over the world” in the classic sense.
(I’d also describe it as a central example of “gaining the power to design a new world order”, but not a central example of “gaining the power to design a new world order from scratch”.)
I think the existence of Hell is incredibly morally relevant and also very game-theoretically relevant too. I probably don’t need to elaborate on the morality side, but for the game theory: IIUC, Keltham really would prefer the universe be destroyed (with him in it!) than that Hell continue to exist. This is different from the stereotypical doomsday threat, which is made by someone who doesn’t actually want doomsday to happen but is hoping that other people will fear it even more and cave.
Yes, I agree. However, as I mentioned in my OP, I think that the prominence of Hell in stories like Unsong and Project Lawful is partly due to them functioning as plot devices to make taking over the world not just ethical but in fact morally obligatory.
Analogously, if a bunch of 19th-century Marxist fiction featured working conditions far harsher than any that existed in the real world, which compelled the heroes to launch a proletarian revolution, you wouldn’t just think “this makes total sense given the fictional premise”, you’d also think “the fictional premise was chosen to help the authors make the thing they already supported (and wanted to write about) seem morally good”.
And “take over the world for good reasons” was IIRC MIRI’s actual plan (hidden under the terminology “decisive strategic advantage” or “pivotal act” or similar).
Ok, but factory farms really do exist? And I don’t think it’s an exaggeration to call them hellish, or hells-on-earth.
And most of the beings that live in our civilization, depending on how you count, live and die in those factory farms.
So it seems that our world is not very different from that of Unsong or Project Lawful, in this respect? Even if the authors chose that setting for literary convenience, and it is only accidentally reflective of reality.
Makes sense yeah.
Still though, I agree with Raemon’s characterization of the Planecrash story.
I’d say ratfics are more about becoming God, and as God you can naturally Fix the world. So you can view rats as atheists who believe that since God doesn’t exist, we must build Him.
Edit: Really, ratfics are about becoming more you are, with becoming God as the natural limit.
This phrase reminds me of a Russian sci-fi piece literally named Hard to Be a God. I expect this piece to be relevant, but I find it hard to explain the relevance without spoilers.
[Haven’t read those fictions or the quoted spoilers] Glad to hear you saying this. In fact, I’ve had a couple quite concerning conversations with a couple different people in which I’m like “so what would you do if you accidentally invented a Friendly AGI in your basement one day? what are some of your important first actions?” and they don’t give one of the incredibly obvious and important answers, and don’t even necessarily agree to it after I say it.
I’ll add that the HM attitude extends to various other things. For example, some people around here are contemptuous of bioethics. I can see where they’re coming from, but I think the attitude is quite wrong, and in particular, one does want to “bring in many stakeholders to the conversation as coequal voices”, because that’s how you be non-HM. Cf. https://www.lesswrong.com/posts/yH9FtLgPJxbimamKg/genomic-emancipation-contra-eugenics
I think it’s quite shameful how rationalists aim to optimize the world and yet are generally uninterested in what life is like for many kinds of normal people, or what problems they face. To rule well, you should know what life is like for your subjects, and what they care about, at minimum.
But there’s also the pattern where “if your users tell you about a problem, they’re always right, and if they tell you the solution, they’re almost always wrong.” And many common forms of “consulting all the stakeholders” ends up giving a veto to special interests, which ends up strangling liberty.
There are some to my knowledge unsolved problems here.
This is still totally High Modernist. To actually rule well you should actually not try to rule as much and should actually try to share power, which includes giving up power. Not giving up all power, but, you know, you leave after your second term in office, and you create a parliament.
I agree on all counts.
I do wonder how much of this is “contemptuous of bioethics done badly”. I can see the argument for “that means we need to do bioethics well, not continue to cede ground” but I do think it’s important to be honest about when fields are failing.
I agree, naturally. I criticize bioethicists as a group, precisely because they are, for the most part, AFAICT, failing to lead on moral questions around reprogenetics.
However, I think one has to make multiple updates. That observation indicates that bioethicists may not have the right abilities or motivations or other properties (determination, grit, courage, sanity, wisdom, what have you), and on those grounds could be dismissed as individuals or even as a group; but it also indicates that the problems themselves are especially difficult. The latter is often underappreciated. To avoid “HM bioethics” you actually have to cede power in the discussions, and even logistically doing that is difficult (I mean, I don’t know how to do it; I don’t have a near-fully-satisfying theory of how to give proper / coequal decision weight to all the stakeholders who should have that).
I think part of the HM mindset is precisely reacting to “a bunch of other people are doing it bad and kinda punishing me for trying to do it better” with “actually I should just be in charge and not worry about the concerns those people talk about” rather than “some other group of people would have to figure out how to do it better”.
HM?
High Modernist
I was going to say that this seems like another one of Eliezer’s founder effects, but he actually wrote about not trusting humans with too much power, in Creating Friendly AI 1.0 (2001):
I wonder how to interpret e.g. the ending of HPMOR in light of this.
Spoiler
HJPEV is bound by a magical oath that prevents this human failing in the same way it is prevented in an agent that meets tiling desiderata. This is explicit in the text. E-Book draft, 2015, chapter 113.
Admittedly this both assumes that the “time of peril” hypothesis is correct and can be handled while maintaing human freedom, and the solution only (in maximum robustness) binds until the end of this time.
My understanding of HPMOR is limited as I’ve only read a few chapters, but looking up the text you cite, it doesn’t seem to prevent most forms of abuse of power.
I always liked this as a fun mini-ratfic, and it doesn’t fall into this pattern, despite the humorously extreme Mary-Sue-ness of its protagonist: https://www.lesswrong.com/posts/LYXb2fLkGDRXoAx7M/timothy-chu-origins-chapter-1
On the other hand, 3 Worlds Collide is an interesting case study:
The first ending is essentially a “use godlike tech-powers to optimize the world” scenario, except that it’s being carried out by the superhappies, rather than a human protagonist. The superhappies do actually care about human wellbeing and try to compromise with human values to the extent they feel possible. Then the second ending basically says, “yeah, regardless of the intended-benevolence of the superhappies, it’s worth blowing up an inhabited star system to avoid being ruled by that kind of god”.
Your overall point seems true and important. The virtues that are enshrined by the rat-fic tradition are primarily “sanity in the midst of insanity (eg. social decoupling and thinking for yourself)” and “heroic responsibility”. These attitudes are not strictly counter to cooperation or working with the existing systems of the world, but they sure do tend to push in the opposite direction. The more you distrust the world, the less you think that you should cooperate with it.
And further, actual practice, “thinking for yourself” directly cuts against human political coordination, since political coalitions almost always maintain unity by coordinating their beliefs, and challenging those coordination beliefs undermines the coalition.
(I find it amusing when people sometimes say that the “one job” of rationalists is to be able to coordinate. I think that’s ass-backwards. The rationalist tradition is about prioritizing independence of thought and the epistemology to discern the truth at the expense of human coordination. There’s some hope there’s another even stronger mode of coordination on the other side of the uncanny valley, but that, so far, remains an unvalidated hope.[1])
However, with regards to this specific point...
Eliezer has said that one of the reasons he writes fanfiction, is that he doesn’t have to invent the world. All of the horror and badness was already present in the source material.
Indeed, a hope that I personally share. But think the folks who are inclined to say that coordination is the “one job”, or similar, of rationalists are missing the plot. Coordinating groups larger than 50 people to accomplish a goal or a political change, by way of accurate views instead of adaptive stories and taboo beliefs, is an unsolved challenge.
I’ll note that overall, the EAs seem to be doing somewhat better than the rationalists at working in groups to get stuff done, and also, they are relatively less free-thinking, and have more taboo beliefs.
A large majority of fantasy settings don’t have literal hells as a key component, so I think my point is still applicable to Project Lawful if you replace “design them that way” with the more general “select for that trait”.
I do agree that this is a good point with regards to HPMoR, which is one reason why I didn’t include HPMoR in my original list of examples.
I don’t think Unsong fits the pattern.
Aaron doesn’t take over the world alone. He merges with seven other wildly different minds, including the villainous Dylan Alvarez. “In William Blake’s prophecies, Albion was the entity formed at the end of time, when all of the different aspects of the human soul finally came together to remake the world”, as one of them says.
And I don’t think the ending is about recreating the world as some kind of rationalist utopia (how would you do that with Dylan an Erica on the team?) - I interpret it more as a “cycle continues” ending where they carry forward God’s already perfect plan into a new world.
See for example this point in the Tosefta, where Scott explains all the Easter eggs:
“As for THARMAS, seven of the ten towers were smoking ruins; the other three were heavily scarred. In the epilogue, THARMAS is going to be used to make the new universe. Seven of ten towers destroyed plus the rest damaged = seven of ten sephirot cracked plus the rest damaged, indicating the new universe will work the same as our own.”
I messaged the team on intercom yesterday and images are now properly spoilered
I basically agree with what you notice, and think that this is what you’d expect if rationalists were mostly normal relative to other people in their goals, which are mostly selfish and dicatorial, but are more intelligent and can think farther ahead about what instrumental goals that they imply for their terminal goals.
Or put another way, the thing that rationalists are doing here are things lots of other people would likely do if they were more intelligent, and the truth of the matter is that most people just like all-powerful dictatorships, almost no matter their ideology.
This is pretty straightforwardly not true, there are plenty of academics (for example) who are as smart as rationalists but don’t do very broad instrumental reasoning.
There are also plenty of people who don’t fantasize about becoming all-powerful dictators.
I think that the hunger to become god is an unusually rationalist trait. Honestly it’s somewhat reminiscent of sociopathy, but fortunately few rationalists seem to be sociopaths. However, I do think a sufficient level of fear of death causes some overlapping traits, e.g. a mentality in which more power is crucial to solving problems. (This is not meant in a particularly blame-y way, I’m just as much an example of this as anyone else around here.)
Fair point, I was generalizing too much here.
I agree with the literal claim that plenty of people don’t fantasize about becoming all-powerful dictators, but I’d say the percentage of people who don’t fantasize (including in their heads and not speaking about it) becoming dictators or don’t believe that an all-powerful dictator is necessary to solve problems/have a good future is much closer to 25-30% than 90% or more here, and this is more of an upper bound than a lower bound.
The reasons for why this is the case partially delve into politics that would cause way more heat than light if I discussed it on here, but one of the reasons for this is that for a lot of citizens, they don’t want to get involved in politics and want someone else to solve their problems for them, and one of the unique traits of a lot of non-dictatorial systems of government is that the average person has to be more involved with politics, and lots of people hate doing this.
An all-powerful dictator where average citizens make none of the decisions in a new world order doesn’t require them to pay attention to their government/politicians, and a lot of people genuinely want the ability to not care about politics at all.