When I first read Hannu Rajaniemi’s Quantum Thief trilogy c. 2015 I had two reactions: delight that this was the most my-ingroup-targeted series I had ever read, and a sinking feeling that ~nobody else would really get it, not just the critics but likely also most fans, many of whom would round his carefully-chosen references off to technobabble. So I was overjoyed to recently find Gwern’s review of it, which Hannu affirms “perfectly nails the emotional core of the trilogy and, true to form, spots a number of easter eggs I thought no one would ever find”, in particular the first few passages:
Stylistically, QT is set relentlessly in media res: neither we nor le Flambeur know why he is in prison, and little is explained thereafter. Hannu makes no concessions to the casual reader, as he mainlines straight into his veins the pre-deep-learning 2010-era transhumanist zeitgeist via Silicon Valley—if it was ever discussed in a late-night bull session after a Singularity University conference, it might pop up here. Hannu crams the novels with blink-and-you’ll-miss-it ideas on the level of Olaf Stapeldon. A conventional Verne gun like Gerald Bull’s is too easy a way of getting to space—how about beating Project Orion by instead using a nuclear space gun (since emulated brains don’t care about high g acceleration)? Or for example, the All-Defector reveals that, since other universes could be rewriting their rules to expand at maximum speed, erasing other universes before they know it, he plans to rewrite our universe’s rule to do so first (ie. he will defect at the multiversal level against all other universes); whereas beginner-level SF like The Three Body Problem would dilate on this for half a book, Hannu’s grand reveal gets all of 2 paragraphs before crashing into the eucatastrophic ending.
For world-building, he drops neologisms left and right, and hard ones at that—few enough American readers will be familiar with the starting premise of “Arsène Lupin in spaaaace!” (probably more are familiar with the anime Lupin The Third these days), but his expectations go far beyond that: the ideal reader of the trilogy is not merely one familiar with the Prisoner’s Dilemma but also with the bizarre zero-determinant PD strategies discovered ~2008, and not just with such basic physics as quantum entanglement or applications like quantum dots, but exotic applications to quantumauctions & game theory (including Prisoner’s Dilemma) & pseudo-telepathy (yes, those are things), and it would definitely be helpful if that reader happened to also be familiar with Eliezer Yudkowsky’s c. 2000s writings on “Coherent Extrapolated Volition”, with a dash of Nikolai Fyodorovich Fyodorov’sRussian Cosmism for seasoning (although only a dash2).
This leads to an irony: I noted while reading Masamune Shirow’s Ghost in the Shell cyberpunk manga that almost everything technical in the GitS manga turned out to be nonsense despite Shirow’s pretensions to in-depth research & meticulous attention to detail in his self-congratulatory author notes; while in QT, most technical things sound like cyberpunk nonsense (and Hannu doesn’t defend them), but are actually real and just so arcane you haven’t heard of them.
For example, some readers accuse Hannu of relying on FTL communication via quantum entanglement, which is bad physics; but Hannu does not! If they had read more closely (similar to the standard reader failure to understand the physics of “Story of Your Life”), they would have noticed that at no point is there communication faster-than-light, only coordination faster-than-light—‘spooky action at a distance’3 He is instead employing advanced forms of quantum entanglement which enable things like secret auctions or for coordinated strategies of game-playing. He explains briefly that the zoku use quantum entanglement in these ways, but a reader could easily miss that, given all the other things they are trying to understand and how common ‘quantum woo’ is.4
The parts of the science I understand were all quite plausible (mind duplication/fractioning and motivations for doing so).
Beyond the accuracy of the science, this was one of the most staggeringly imaginative and beautifully written scifi books I’ve ever read. It’s for a very particular audience, but if you’re here you might be that audience. If you are, this might be the best book you’ve read.
Attention conservation notice: 3,000+ words of longform quotes by various folks on the nature of personal identity in a posthuman future, and hiveminds / clans
As an aside, one of the key themes running throughout the Quantum Thief trilogy is the question of how you might maintain personal identity (in the pragmatic security sense, not the philosophical one) in a future so posthuman that minds can be copied and forked indefinitely over time. To spoil Hannu’s answer:
… Jean & the Sobornost Founders & the zoku elders are all defined by what, at their core, they want. Anyone who wants the same thing is, for all (their) intents and purposes, the same person as them; because they want the same unchanging things, they can be trusted as the original. The ‘Founder codes’, and Jean’s final password to unlock his sealed memories, are all memories of what defines their wants: the Founder Sumanguru wants blood & fire & electricity & screaming children, and enemies to destroy; the Founder Chen recall the trauma of livestreaming their father’s assassination, remaining eternally resolved that the last enemy that shall be defeated is death; while seared into the minds of the Founder Joséphine Pellegrinis is the final thought of their founder, her desperate dying wish that her lover Jean le Flambeur someday return to her… (And the zoku elders want to empower their zoku clans.)
But even personal identity frays under the power of time: given freedom to change, sooner or later, like the Ship of Theseus, the mind which sets out is not the mind which arrives. So the price of immortality must be that one cannot change: one is condemned to want the same things, forever.7 (“There is no prison, except in your mind.”) Joséphine Pellegrini cannot stop seeking after her lost Jean—nor can Jean stop his thieving nor trying to escape her, because le Flambeur, what does Jean le Flambeur remember?
I take Anders Sandberg’s answer to be on the other end of this spectrum; he doesn’t mind changing over time such that he might end up wanting different things:
Anders Sandberg: I think one underappreciated thing is that if we can survive for a very long time individually, we need to reorganise our minds and memories in interesting ways. There is a kind of standard argument you sometimes hear if you’re a transhumanist — like I am — that talks about life extension, where somebody cleverly points out that you would change across your lifetime. If it’s long enough, you will change into a different person. So actually you don’t get an indefinitely extended life; you just get a very long life thread. I think this is actually an interesting objection, but I’m fine with turning into a different future person. Anders Prime might have developed from Anders in an appropriate way — we all endorse every step along the way — and the fact that Anders Prime now is a very different person is fine. And then Anders Prime turns into Anders Biss and so on — a long sequence along a long thread.
(I have mixed feelings about Anders’ take: I have myself changed so profoundly since youth that that my younger self would not just disendorse but be horrified by the person I am now, yet I did endorse every step along the way, and current-me still does upon reflection (but of course I do). Would current-me also endorse a similar degree of change going forward, even subject to every step being endorsed by the me right before change? Most likely not, perhaps excepting changes towards some sort of reflective equilibrium.)
I interpret Holden Karnofsky’s take to be somewhere in between, perhaps closer to Hannu’s answer. Holden remarked that he doesn’t find most paradoxical thought experiments about personal identity (e.g. “Would a duplicate of you be “you?”″ or “If you got physically destroyed and replaced with an exact duplicate of yourself, did you die?”) all that confounding because his personal philosophy on “what counts as death” dissolves them, and that his philosophy is simple, comprising just 2 aspects: constant replacement (“in an important sense, I stop existing and am replaced by a new person each moment”) and kinship with future selves. Elaborating on the latter:
My future self is a different person from me, but he has an awful lot in common with me: personality, relationships, ongoing projects, and more. Things like my relationships and projects are most of what give my current moment meaning, so it’s very important to me whether my future selves are around to continue them.
So although my future self is a different person, I care about him a lot, for the same sorts of reasons I care about friends and loved ones (and their future selves).3
If I were to “die” in the common-usage (e.g., medical) sense, that would be bad for all those future selves that I care about a lot.4
…
[One of the pros of this view]
It seems good that when I think about questions like “Would situation __ count as dying?”, I don’t have to give answers that are dependent on stuff like how fast the atoms in my body turn over—stuff I have basically never thought about and that doesn’t feel deeply relevant to what I care about. Instead, when I think about whether I’d be comfortable with something like teleportation, I find myself thinking about things I actually do care about, like my life projects and relationships, and the future interactions between me and the world.
Rob Wiblin: … one of the non-AI blog posts you’ve written, which I really enjoyed reading this week when I was prepping for the conversation, is called Characterising utopia. … Some of the shifts that you envisaged wouldn’t be super surprising. Like we could reduce the amount that people experience physical pain, and we could make people be a lot more energetic and a lot more cheerful. But you had a section called “Contentious changes.” What are some of the contentious changes, or possible changes, that you envisage in a utopia?
Richard Ngo: One of the contentious changes here is to do with individualism, and how much more of it or less of it we have in the future than we have today. Because we’ve been on this trend towards much more individualistic societies, where there are fewer constraints on what people do that are externally imposed by society.
I could see this trend continuing, but I could also see it going in the opposite direction. Maybe, for example, in a digital future, we’ll be able to make many copies of ourselves, and so this whole concept of my “personal identity” starts to shift a little bit and maybe I start to think of myself as not just one individual, but a whole group of individuals or this larger entity. And in general, it feels like being part of a larger entity is really meaningful to people and really shapes a lot of people’s lives, whether that’s religion, whether that’s communities, families, things like that.
The problem historically has just been that you don’t get to choose it — you just have to get pushed into this entity that maybe isn’t looking out for your best interests. So it feels interesting to me to wonder if we can in fact design these larger entities or larger superorganisms that are really actually good for the individuals inside, as well as providing this more cohesive structure for them. Is that actually something we want? Would I be willing to lose my individuality if I were part of this group of people who were, for example, reading each other’s minds or just having much less privacy than we have today, if that was set up in such a way that I found it really fulfilling and satisfying?
I really don’t know at all, but it seems like the type of question that is really intriguing and provides a lot of scope for thinking about how technology could just change the ways in which we want to interact with each other.
Rob Wiblin: I’m so inculcated into the individualist culture that the idea slightly makes my skin crawl thinking about any of this stuff. But I think if you tried to look objectively at what has caused human wellbeing throughout history, then it does seem like a somewhat less individualistic culture, where people have deeper ties and commitments to one another, maybe that is totally fine — and I’ve just drunk the Kool-Aid thinking that being an atomised individual is so great.
Richard Ngo: If you know the book, The WEIRDest People in the World, which describes the trend towards individualism and weaker societal ties, I think the people in our circles are the WEIRDest people of the WEIRDest people in the world — where “WEIRD” here is an acronym meaning “Western, educated, industrialised, rich, and democratic,” not just “weird.” So we are the WEIRDest people of the WEIRDest countries. And then you’re not a bad candidate for the WEIRDest person in the WEIRDest community in the WEIRDest countries that we currently have, Rob. So I’m not really too surprised by that.
(I thought it was both interesting and predictable that Rob would find the idea discomfiting; coming from a non-WEIRD culture, I found Richard’s idea immediately attractive and aesthetically “right”.)
Richard gives an fictional example of what this might look like from a first-person perspective in his recent short story The Gentle Romance—if you’re reading this Richard, do let me know if you want this removed:
As ze reconnects more deeply with zir community, that oceanic sense of oneness arises more often. Some of zir friends submerge themselves into a constant group flow state, rarely coming out. Each of them retains their individual identity, but the flows of information between them increase massively, allowing them to think as a single hivemind. Ze remains hesitant, though. The parts of zir that always wanted to be exceptional see the hivemind as a surrender to conformity. But what did ze want to be exceptional for? Reflecting, ze realizes that zir underlying goal all along was to be special enough to find somewhere ze could belong. The hivemind allows zir to experience that directly, and so ze spends more and more time within it, enveloped in the warm blanket of a community as close-knit as zir own mind.
Outside zir hivemind, billions of people choose to stay in their physical bodies, or to upload while remaining individuals. But over time, more and more decide to join hiveminds of various kinds, which continue to expand and multiply. By the time humanity decides to colonize the stars, the solar system is dotted with millions of hiveminds. A call goes out for those willing to fork themselves and join the colonization wave. This will be very different from anything they’ve experienced before — the new society will be designed from the ground up to accommodate virtual humans. There will be so many channels for information to flow so fluidly between them that each colony will essentially be a single organism composed of a billion minds.
Ze remembers loving the idea of conquering the stars — and though ze is a very different person now, ze still feels nostalgic for that old dream. So ze argues in favor when the hivemind debates whether to prioritize the excitement of exploration over the peacefulness of stability. It’s a more difficult decision than any the hivemind has ever faced, and no single satisfactory resolution emerges. So for the first time in its history, the hivemind temporarily fractures itself, giving each of its original members a chance to decide on an individual basis whether they’ll go or stay.
I think Richard’s notion of ‘hivemind’ is cousin to Robin Hanson’s ‘clan’ from Age of Em (although unlike Richard’s lovely story, Hanson’s depiction of an em-filled future has never stopped seeming dystopian to me, Hanson’s protestation to the contrary that “[readers repelled by aspects of the em era should] try hard to see this world from its residents’ point of view, before disowning these their plausible descendants”, albeit far more granular, comprehensive and first-principles-based):
The set of all em copies of the same original human constitutes a “clan.” Most wages go to the 1000 most productive clans, who are each known by one name, like “John,” who know each other very well, and who discriminate against less common clans. Compared with people today, ems are about as elite as billionaires, heads of state, and Olympic gold medalists. The em world is more competitive than ours in more quickly eliminating less productive entities and practices. This encourages more job punishment, less product variety and identity enhancement, and more simple functionality. Because they are more productive, ems tend to be married, religious, smart, gritty, mindful, extraverted, conscientiousness, agreeable, non-neurotic, and morning larks.
Many myths circulate about factors that increase economic growth rates. For example, the fact that ems can run faster than humans should not much increase growth. Even so, the em economy grows faster than does ours because of stronger competition, computers mattering more, and especially because factories can make labor as fast as non-labor capital. An em economy doubling time estimate of a few weeks comes from the time for factories to duplicate their mass today, and from the historical trend in growth rates. In response, capital becomes less durable, and one-time-use products become more attractive. Clans become a unit of finance, private firms and hostile takeovers get more support, and asset prices more closely approximate the predictions derived from strong financial competition.
Ems trust their clans more than we trust families or identical twins. So clans are units of finance, liability, politics, labor negotiations, and consumer purchasing. To promote unity, clans avoid members arguing or competing. Em firms are larger, better managed, put more effort into coordination, have more specific job roles, focus more on costs relative to novelty, and have higher market shares and lower markups. Clan reputations and clans buying into firms promotes clan-firm trust, which supports locating employees at firms, using distinctive work styles, and focusing more on being useful instead of gaming firm evaluation systems. Em work teams tend to have similar social-category features like age but a diversity of information sources and thinking styles. In mass-labor markets, ems are created together, end or retire together, almost never break up, and mostly socialize internally. In niche-labor markets, associates coordinate less regarding when they are created or retire.
Faster ems have many features that mark them as higher status, and the clumping of speeds creates a class system of distinct status levels. Strong central rulers are more feasible for ems, as leaders can run faster, put spurs in high-trust roles, and use safes to reassure wary citizens. Decision markets can help advise key government decisions, while combinatorial auctions can help to make complex interdependent allocations. The em world selects for personalities good at governing that same personality. Competitive clans and cities may commit to governing via decision markets that promote profit or long-term influence. One em one vote works badly, but speed-weighted voting seems feasible, although it requires intrusive monitoring. Shifting coalitions of em clans may dominate the politics of em firms and cities, inducing high costs of lobbying and change. Ems may try many policies to limit such clan coalition politics.
As ems don’t need sex to reproduce, sex is left more to individual choice, and may be suppressed as in eunuchs. But demand for sex and romantic pair-bonding likely persists, as do many familiar gendered behavioral patterns. A modestly unequal demand for male versus female workers can be accommodated via pairs whose partners run at different speeds, or who use different ratios of spurs to other workers. Ems have spectacularly good looks in virtual reality, and are very accomplished. Open-source em lovers give all ems an attractive lower bound on relation quality. Clan experience helps ems guess who are good receptive matches. Having only one em from each clan in each social setting avoids complicating relations.
Ems show off their abilities and loyalties, although less than we do because ems are poorer and better-known to each other. Because speed is easy to pay for, ems show off more via clever than fast speech. Celebrities matter less to ems, and it is easy to meet with a celebrity, but hard to get them to remember you. Clans coordinate to jointly signal shared features like intelligence, drive, and fame. Clans fund young ems to do impressive things, about which many older copies can brag. Innovation may matter less for em intellectuals. Mind-theft inspires great moral outrage and charity efforts. Secure in identifying with their clan, most ems focus personal energy more on identifying with their particular job, team, and associates. It isn’t clear if em identity degrades continuously or discretely as copies get more different. Copy-events are identity-defining, and newly copied teams quickly create distinct team cultures.
Ems are likely to reverse our recent trend away from religion and overt rituals, perhaps via more group singing. Traditional religions can continue, but need doctrinal clarifications on death and sins of copies. Like high stress workers today, em work groups pushed to their limits swear, insult, and tease. Ems deal with a wider range of mind opacity and transparency, allowing mind reading within teams, but manipulating expressions to hide from outsiders. Clans can offer members life-coaching via voices in their heads, using statistics from similar copies, but teams may create unique cultures which limit the usefulness of that. Avoiding direct meetings helps clans bond better. Em relations are often in the context of similar relations between copies. At work, ems try more to make relations similar, to gain from learning and scale economics. But friends keep relations more different, to emphasize loyalty and natural feelings.
Em culture emphasizes industriousness, work and long-term orientations, and low context attitudes toward rules and communication. Being poorer, ems tend towards farmer/conservative values, relative to forager/liberal values. So ems more value honor, order, hierarchy, religion, work, and less value sharing, consensus, travel, leisure, and variety. Sex attitudes stay more forager-like, however. Ems are divided like we are by geographic region, young versus old, male versus female, rich versus poor, and city center versus periphery. Ems also divide by varying speeds, physical versus virtual work, remembering the human era versus not, and large versus small clans. Ems travel to visit or swap with other copies of themselves. An exotic travel destination is other speed cultures. Like us, ems tell stories of conflict and norm violations, set in ancestral situations. Stories serve as marketing, with many characters coming from well-known clans. Em stories have less death and fast-action.
The short story The Epiphany of Gliese 581 by Fernando Borretti has something of the same vibe as Rajaniemi’s QT trilogy; Borretti describes it as inspired by Orion’s Arm and the works of David Zindell. Here’s a passage describing a flourishing star system already transformed by weakly posthuman tech:
The world outside Susa was a lenticular cloud of millions of lights, a galaxy in miniature, each a world unto itself. There were clusters of green lights that were comets overgrown with vacuum trees, and plant and animal and human life no Linnaeus would recognize. There were points of dull red light, the reversible computers where bodyless people lived. And there were arcs of blue that were ring habitats: ribbons tied end-to-end, holding concave ocean, and the oceans held continents, islands, mountain ranges, rivers, forests and buried ruins, endless forms of life, cities made of glass, paradise regained. All this had been inanimate dust and cratered wasteland, which human hands had made into an oasis in the sky, where quadrillions live who will never die.
The posthumans who live there called it Ctesiphon. And at times they call it paradise, after the Persian word for garden.
And at the center of the oasis there was a star that travelled backwards across the H-R diagram: already one one-hundredth of it had been whittled away; made into a necklace of artificial gas giants in preparation for the end of time; or sent through reactors where disembodied chemists made protons into carbon, oxygen, lithium and sodium, the vital construction material. And in time nothing would be left but a dim red ember encircled by cryojovian fuel depots. And the habitats would be illuminated by electric diodes.
Another star system, this time still being transformed:
Wepwawet was a dull red star, ringed by water droplets the size of mountains, where some two hundred billion people lived who breathed water. There was a planet made of stone shrouded in steam, and a train of comets, aimed by human hands from beyond the frostline, delivered constant injections of water. When the vapour condensed there would be ocean, and the shapers would get to work on the continents. Other Earths like this had been cast, like seeds, across the entire breadth of the cosmos.
The system was underpopulated: resources were abundant and people were few, and they could bask in the sun and, for a time, ignore the prophecies of Malthus, whose successors know in time there won’t be suns.
This was the first any of them had seen of nature. Not the landscaped, continent-sized gardens of Ctesiphon, where every stone had been set purposefully and after an aesthetic standard, but nature before human hands had redeemed it: an endless, sterile wasteland. The sight of scalding, airless rocks disturbed them.
When I first read Hannu Rajaniemi’s Quantum Thief trilogy c. 2015 I had two reactions: delight that this was the most my-ingroup-targeted series I had ever read, and a sinking feeling that ~nobody else would really get it, not just the critics but likely also most fans, many of whom would round his carefully-chosen references off to technobabble. So I was overjoyed to recently find Gwern’s review of it, which Hannu affirms “perfectly nails the emotional core of the trilogy and, true to form, spots a number of easter eggs I thought no one would ever find”, in particular the first few passages:
The parts of the science I understand were all quite plausible (mind duplication/fractioning and motivations for doing so).
Beyond the accuracy of the science, this was one of the most staggeringly imaginative and beautifully written scifi books I’ve ever read. It’s for a very particular audience, but if you’re here you might be that audience. If you are, this might be the best book you’ve read.
Attention conservation notice: 3,000+ words of longform quotes by various folks on the nature of personal identity in a posthuman future, and hiveminds / clans
As an aside, one of the key themes running throughout the Quantum Thief trilogy is the question of how you might maintain personal identity (in the pragmatic security sense, not the philosophical one) in a future so posthuman that minds can be copied and forked indefinitely over time. To spoil Hannu’s answer:
I take Anders Sandberg’s answer to be on the other end of this spectrum; he doesn’t mind changing over time such that he might end up wanting different things:
(I have mixed feelings about Anders’ take: I have myself changed so profoundly since youth that that my younger self would not just disendorse but be horrified by the person I am now, yet I did endorse every step along the way, and current-me still does upon reflection (but of course I do). Would current-me also endorse a similar degree of change going forward, even subject to every step being endorsed by the me right before change? Most likely not, perhaps excepting changes towards some sort of reflective equilibrium.)
I interpret Holden Karnofsky’s take to be somewhere in between, perhaps closer to Hannu’s answer. Holden remarked that he doesn’t find most paradoxical thought experiments about personal identity (e.g. “Would a duplicate of you be “you?”″ or “If you got physically destroyed and replaced with an exact duplicate of yourself, did you die?”) all that confounding because his personal philosophy on “what counts as death” dissolves them, and that his philosophy is simple, comprising just 2 aspects: constant replacement (“in an important sense, I stop existing and am replaced by a new person each moment”) and kinship with future selves. Elaborating on the latter:
Richard Ngo goes in a different direction with the “personal identity in a posthuman future” question:
(I thought it was both interesting and predictable that Rob would find the idea discomfiting; coming from a non-WEIRD culture, I found Richard’s idea immediately attractive and aesthetically “right”.)
Richard gives an fictional example of what this might look like from a first-person perspective in his recent short story The Gentle Romance—if you’re reading this Richard, do let me know if you want this removed:
I think Richard’s notion of ‘hivemind’ is cousin to Robin Hanson’s ‘clan’ from Age of Em (although unlike Richard’s lovely story, Hanson’s depiction of an em-filled future has never stopped seeming dystopian to me, Hanson’s protestation to the contrary that “[readers repelled by aspects of the em era should] try hard to see this world from its residents’ point of view, before disowning these their plausible descendants”, albeit far more granular, comprehensive and first-principles-based):
The short story The Epiphany of Gliese 581 by Fernando Borretti has something of the same vibe as Rajaniemi’s QT trilogy; Borretti describes it as inspired by Orion’s Arm and the works of David Zindell. Here’s a passage describing a flourishing star system already transformed by weakly posthuman tech:
Another star system, this time still being transformed: