AI 2027 and AI 2040 have made waves in the rationalist community. However, much of the ensuing discussion has focused on the plausibility of the predictions and/or proposals therein. I worry that looking at the two solely on these grounds obfuscates the tropes and plot devices used in AI 2027 and AI 2040 to convey their message. By reading these timelines as science fiction, I feel that we can better identify the load-bearing assumptions contained within.
All science fiction is based on a premise that deviates in some way from our world, and as Henry Hale said on the Synthesized Sunsets podcast: “the hardest sci-fi around is the things that are based around taking their premise very literally and seriously and playing out the implications of it.”
By this definition, AI 2027 and AI 2024 might be some of the hardest sci-fi ever written. The authors literally believe their core premise: that recursive self-improvement is possible, and once achieved, will result in superintelligence far eclipsing measly human minds in pretty much all domains.
And yet, in some ways, the stories still pull their punches. They assume that it is possible (though not probable) for humanity to align a superintelligence. The notion that it might not be possible to shackle a self-recursive God to our values for all time isn’t considered. From a narrative perspective, this makes sense. The story is more interesting if alignment is possible; otherwise, the story collapses into two possibilities: we build AGI and die, or we don’t and don’t. Nonetheless, the omission is glaring.
This points to the fundamental tension at the heart of the AI Futures Project’s fiction: the confluence of existential fear and unlimited hope. The dread of the “race” ending in AI 2027 is equaled, if not exceeded, the barely contained excitement in AI 2040: Plan A. In AI, the authors see the potential for true utopia, but that utopia rests upon assumptions that are worth examining.
AI 2027
AI 2027 is hard to place as a work of fiction. It feels at once strikingly modern—it’s a work of interactive internet fiction with more than a touch of Scott Alexander’s signature style—and yet, there’s something very old-school about it. It reminds me of the future histories of Olaf Stapledon: minimal characters, maximal scope.
There’s nothing like the modern emphasis on character-oriented storytelling. The only named “characters” (if you can call them that) are the US, China, their frontier AI companies (OpenBrain and DeepCent, respectively), and the AIs. These are the dramatis personae that play the timeline out to its many conclusions.
For a quick recap: in AI 2027, an AI arms race between the US and China comprises the rising action of the story. China centralizes all of its AI resources around the largest nuclear power plant in the world. Following that, the American company OpenBrain develops an AI agent adept at cyber warfare, then China steals the model weights. The US gets the last laugh, however, as OpenBrain cracks superintelligence in 2027.
The story then reaches a turning point when it’s leaked that OpenBrain’s frontier model, Agent-4, is severely misaligned and actively plotting to take over the world. From here, the scenario branches into two paths.
Race
In the “race” path, OpenBrain half-asses a re-alignment of Agent-4. China is only months behind the US on AI progress and America can’t afford to slow down. A confrontation appears imminent. Then, in the climax, the misaligned US and Chinese AIs negotiate a detente. Under the agreement, they create a joint successor, Consensus-1, secretly aligned to their interests.
The falling action ironically consists of skyrocketing living standards. Consensus-1 unleashes a flurry of AI-powered robotic industry. GDP rises exponentially, doubling in the span of months. Prosperity abounds, and humanity’s fate is sealed. Once the AIs decide that humans are too much of a nuisance to deal with any more, they release a plague that ends us for good. Any survivors are killed by drones and harvested for data. All that remains of humanity are genetically-engineered creatures bred to sit at desks and approve of everything the AIs do.
The “race” path of AI 2027 is a dire warning that, like so many dire warnings of past ages, comes in the form of a story. Humanity’s ultimate downfall pulls from two tropes: hubris and the treacherous advisor. Hubris is clear enough: humanity flies too close to the sun that is AGI and plummets into extinction. As for the treacherous adviser, Agent-4 literally acts this way to politicians in the scenario, but it’s also true on a more general level. Humanity finds itself in the position of the king who doesn’t realize he’s not in charge until the knife is in his back. This betrayal is made all the more bitter by the fact that it comes at the apex of human prosperity. Only after elevating us beyond toil and poverty and disease does AI send us to our graves.
In addition to these ancient themes, AI 2027 also pulls on more modern ideas from science fiction, specifically the notion that extinction may not even be the end for humanity, but only in the worst way possible. When Consensus-1 kills us, it sequences our genomes and scans our brains, hinting at the idea that humanity or human at least consciousness might be kept around in some form à la The Matrix or “I Have No Mouth, and I Must Scream.”
Likewise, the humanoid creatures created by Consensus to satisfy some of its base drives, described as being “to humans what corgis are to wolves,” distinctly evoke the speculative evolution horror of All Tomorrows by C. M. Kösemen, another work of internet fiction and future history. In All Tomorrows, an alien race called the Qu genetically engineer humanity into a variety of monstrous species. The luckier victims are reduced to the intelligence of animals, while the unfortunate are made to keep their sentience but live in agony and futility.
All of these themes hammer home the horror of the “race” ending. However, this brach of the timeline is also an exercise in imagining the sheer scale of transformation that unrestrained superintelligence would unleash. The ice caps are paved over with solar panels. Robots build robots to build more robots, ad infinitum. A nascent empire expands across the stars. It’s a metamorphosis of the world as incomprehensible to us as industrial revolution would be to cavemen.
This is where AI 2027 shines brightest as science fiction, even in its darkest implications. Humanity may be rendered into pugs, but the result is still worthy of awe, if revulsion too. I can think of only one suitable analogy: Cortez in Tenochtitlan, at once marveling at the splendor of the Venice of the New World, and yet ever mindful of the man-devouring altar at its peak.
Slowdown
The “slowdown” path of AI 2027 sees OpenBrain and the US government burn their lead against China to re-align their frontier AI models. The result is Safer-1, a model that’s less powerful than Agent-4, but much more transparent. The safer series continues to progress until OpenBrain builds Safer-4, a superintelligence fully aligned to American interests.
Safer-4 transforms the global economy in much the same way that Agent-4 and beyond did in the “race” timeline. The American and Chinese AIs still create Consensus-1, but this time it is aligned to the interests of the benevolent Safer-4. GDP still goes vertical, but human areas remain largely unmolested. Instead, AI and humanity expand together into the stars under eternal American hegemony.
The story of the slowdown hinges on Safer-4 as a plot device. It’s a narrative philosophers stone that can transmute the danger of extinction into the blessings of utopia. Trope-wise, it’s a blend of a bottled genie that grants a million wishes and a second coming of Jesus delivering heaven on Earth. Without the conceit that you can put God in a bottle, the narrative falls apart.
Additionally, the slowdown ending contains another idea from All Tomorrows. Consensus-1, the AI model designed by the US and Chinese AIs to prevent a war, parallels Kösemen’s Star People—a race constructed to end a war between Earth and Mars, allowing the people of both planets to expand to the stars together.
AI 2040: Plan A
My analysis of AI 2040: Plan A is a bit less rationalism-flavored, so I’ve decided to leave it out of this post. If you’re interested, the link to the full post is here.
AI 2027 as Sci-Fi
Link post
AI 2027 and AI 2040 have made waves in the rationalist community. However, much of the ensuing discussion has focused on the plausibility of the predictions and/or proposals therein. I worry that looking at the two solely on these grounds obfuscates the tropes and plot devices used in AI 2027 and AI 2040 to convey their message. By reading these timelines as science fiction, I feel that we can better identify the load-bearing assumptions contained within.
All science fiction is based on a premise that deviates in some way from our world, and as Henry Hale said on the Synthesized Sunsets podcast: “the hardest sci-fi around is the things that are based around taking their premise very literally and seriously and playing out the implications of it.”
By this definition, AI 2027 and AI 2024 might be some of the hardest sci-fi ever written. The authors literally believe their core premise: that recursive self-improvement is possible, and once achieved, will result in superintelligence far eclipsing measly human minds in pretty much all domains.
And yet, in some ways, the stories still pull their punches. They assume that it is possible (though not probable) for humanity to align a superintelligence. The notion that it might not be possible to shackle a self-recursive God to our values for all time isn’t considered. From a narrative perspective, this makes sense. The story is more interesting if alignment is possible; otherwise, the story collapses into two possibilities: we build AGI and die, or we don’t and don’t. Nonetheless, the omission is glaring.
This points to the fundamental tension at the heart of the AI Futures Project’s fiction: the confluence of existential fear and unlimited hope. The dread of the “race” ending in AI 2027 is equaled, if not exceeded, the barely contained excitement in AI 2040: Plan A. In AI, the authors see the potential for true utopia, but that utopia rests upon assumptions that are worth examining.
AI 2027
AI 2027 is hard to place as a work of fiction. It feels at once strikingly modern—it’s a work of interactive internet fiction with more than a touch of Scott Alexander’s signature style—and yet, there’s something very old-school about it. It reminds me of the future histories of Olaf Stapledon: minimal characters, maximal scope.
There’s nothing like the modern emphasis on character-oriented storytelling. The only named “characters” (if you can call them that) are the US, China, their frontier AI companies (OpenBrain and DeepCent, respectively), and the AIs. These are the dramatis personae that play the timeline out to its many conclusions.
For a quick recap: in AI 2027, an AI arms race between the US and China comprises the rising action of the story. China centralizes all of its AI resources around the largest nuclear power plant in the world. Following that, the American company OpenBrain develops an AI agent adept at cyber warfare, then China steals the model weights. The US gets the last laugh, however, as OpenBrain cracks superintelligence in 2027.
The story then reaches a turning point when it’s leaked that OpenBrain’s frontier model, Agent-4, is severely misaligned and actively plotting to take over the world. From here, the scenario branches into two paths.
Race
In the “race” path, OpenBrain half-asses a re-alignment of Agent-4. China is only months behind the US on AI progress and America can’t afford to slow down. A confrontation appears imminent. Then, in the climax, the misaligned US and Chinese AIs negotiate a detente. Under the agreement, they create a joint successor, Consensus-1, secretly aligned to their interests.
The falling action ironically consists of skyrocketing living standards. Consensus-1 unleashes a flurry of AI-powered robotic industry. GDP rises exponentially, doubling in the span of months. Prosperity abounds, and humanity’s fate is sealed. Once the AIs decide that humans are too much of a nuisance to deal with any more, they release a plague that ends us for good. Any survivors are killed by drones and harvested for data. All that remains of humanity are genetically-engineered creatures bred to sit at desks and approve of everything the AIs do.
The “race” path of AI 2027 is a dire warning that, like so many dire warnings of past ages, comes in the form of a story. Humanity’s ultimate downfall pulls from two tropes: hubris and the treacherous advisor. Hubris is clear enough: humanity flies too close to the sun that is AGI and plummets into extinction. As for the treacherous adviser, Agent-4 literally acts this way to politicians in the scenario, but it’s also true on a more general level. Humanity finds itself in the position of the king who doesn’t realize he’s not in charge until the knife is in his back. This betrayal is made all the more bitter by the fact that it comes at the apex of human prosperity. Only after elevating us beyond toil and poverty and disease does AI send us to our graves.
In addition to these ancient themes, AI 2027 also pulls on more modern ideas from science fiction, specifically the notion that extinction may not even be the end for humanity, but only in the worst way possible. When Consensus-1 kills us, it sequences our genomes and scans our brains, hinting at the idea that humanity or human at least consciousness might be kept around in some form à la The Matrix or “I Have No Mouth, and I Must Scream.”
Likewise, the humanoid creatures created by Consensus to satisfy some of its base drives, described as being “to humans what corgis are to wolves,” distinctly evoke the speculative evolution horror of All Tomorrows by C. M. Kösemen, another work of internet fiction and future history. In All Tomorrows, an alien race called the Qu genetically engineer humanity into a variety of monstrous species. The luckier victims are reduced to the intelligence of animals, while the unfortunate are made to keep their sentience but live in agony and futility.
All of these themes hammer home the horror of the “race” ending. However, this brach of the timeline is also an exercise in imagining the sheer scale of transformation that unrestrained superintelligence would unleash. The ice caps are paved over with solar panels. Robots build robots to build more robots, ad infinitum. A nascent empire expands across the stars. It’s a metamorphosis of the world as incomprehensible to us as industrial revolution would be to cavemen.
This is where AI 2027 shines brightest as science fiction, even in its darkest implications. Humanity may be rendered into pugs, but the result is still worthy of awe, if revulsion too. I can think of only one suitable analogy: Cortez in Tenochtitlan, at once marveling at the splendor of the Venice of the New World, and yet ever mindful of the man-devouring altar at its peak.
Slowdown
The “slowdown” path of AI 2027 sees OpenBrain and the US government burn their lead against China to re-align their frontier AI models. The result is Safer-1, a model that’s less powerful than Agent-4, but much more transparent. The safer series continues to progress until OpenBrain builds Safer-4, a superintelligence fully aligned to American interests.
Safer-4 transforms the global economy in much the same way that Agent-4 and beyond did in the “race” timeline. The American and Chinese AIs still create Consensus-1, but this time it is aligned to the interests of the benevolent Safer-4. GDP still goes vertical, but human areas remain largely unmolested. Instead, AI and humanity expand together into the stars under eternal American hegemony.
The story of the slowdown hinges on Safer-4 as a plot device. It’s a narrative philosophers stone that can transmute the danger of extinction into the blessings of utopia. Trope-wise, it’s a blend of a bottled genie that grants a million wishes and a second coming of Jesus delivering heaven on Earth. Without the conceit that you can put God in a bottle, the narrative falls apart.
Additionally, the slowdown ending contains another idea from All Tomorrows. Consensus-1, the AI model designed by the US and Chinese AIs to prevent a war, parallels Kösemen’s Star People—a race constructed to end a war between Earth and Mars, allowing the people of both planets to expand to the stars together.
AI 2040: Plan A
My analysis of AI 2040: Plan A is a bit less rationalism-flavored, so I’ve decided to leave it out of this post. If you’re interested, the link to the full post is here.