You have a point in that Vinge portrays outward migration into higher Zones, with all their unexplained advantages including computational advantage, as part of the process by which a civilization of natural intelligences evolves to the point of producing a superintelligence. (For those who haven’t seen the book, the Zones are concentric regions of the galaxy, in which the further out you go, the more advanced the technology that is possible, including superintelligence and faster-than-light travel.)
Mitchell_Porter
Karma: 9,288
I assume Manifold here means “reality”, and not just the betting site?
I don’t know that this would fit with the idea of no free will. Surely you’re not really making any decisions.
This sounds like “epiphenomenalism”—the idea that the conscious mind has no causal power, it’s just somehow along for the ride of existence, while atoms or whatever do all the work. This is a philosophy that alienates you from your own power to choose.
But there is also “compatibilism”. This is originally the idea that free will is compatible with determinism, because free will is here defined to mean, not that personal decisions have no causes at all, but that all the causes are internal to the person who decides.
A criticism of compatibilism is that this definition isn’t what’s meant by free will. Maybe so. But for the present discussion, it gives us a concept of personal choice which isn’t disconnected from the rest of cause and effect.
We can consider simpler mechanical analogs. Consider any device that “makes choices”, whether it’s a climate control system in a building, or a computer running multiple processes. Does epiphenomenalism make sense here? Is the device irrelevant to the “choice” that happens? I’d say no: the device is the entity that performs the action. The action has a cause, but it is the state of the device itself, along with the relevant physical laws, which is the cause.
We can think similarly of human actions where conscious choice is involved.
But your values wouldn’t have been decided by you.
Perhaps you didn’t choose your original values. But a person’s values can change, and if this was a matter of self-aware choice between two value systems, I’m willing to say that the person decided on their new values.
AI interpretability can assign meaning to states of an AI, but what about process? Are there principled ways of concluding that an AI is thinking, deciding, trying, and so on?
It would hardly be the first time that someone powerful went mad, or was thought to be mad by those around them, and the whole affair was hushed up, or the courtiers just went along with it. Wikipedia says that the story of the emperor’s new clothes goes back at least to 1335… Just last month, Zvi was posting someone’s theory about why rich people go mad. I think the first time I became aware of the brewing alarm around “AI psychosis” was the case of Geoff Lewis, a billionaire VC who has neither disowned his AI-enhanced paranoia of a few months ago, nor kept going with it (instead he got married). And I think I first heard of “vibe physics” in connection with Uber founder Trevor Kalanick.
The consequences for an individual depend on the details. For example, if you still understand yourself as being part of the causal chain of events, because you make decisions that determine your actions—it’s just that your decisions are in turn determined by psychological factors like personality, experience, and intelligence—your sense of agency may remain entirely unaffected. The belief could even impact your decision-making positively, e.g. via a series of thoughts like “my decisions will be determined by my values”—“what do my values actually imply I should do in this situation”—followed by enhanced attention to reasoning about the decision.
On the other hand, one hears that loss of belief in free will can be accompanied by loss of agency or loss of morality, so, the consequences really depend on the psychological details. In general, I think an anti-free-will position that alienates you from the supposed causal machinery of your decision-making, rather than one that identifies you with it, has the potential to diminish a person.
I have three paradigms for how something like this might “work” or at least be popular:
Filters as used in smartphone photos and videos. Here the power to modify the image takes place strictly as an addendum to the context of real human-to-human communication. The Sora 2 app seems a bit like an attempt to apply this model to the much more powerful capabilities of generative video.
The Sora 1 feed. This is just a feed of images and videos created by users, that other users can vote on. The extra twist is that you can usually see the prompt, storyboard, and source material used to generate them, so you can take that material and create your own variations… This paradigm is that of a genuine community of creators—people who were using Sora anyway, and are now able to study and appropriate each other’s creations. One difference between this paradigm and the “filter” paradigm, is that the characters appearing in the creations are not the users, they are basically famous or fictional people.
Virtual reality / shared gaming worlds. It seems to me that something like this is favorable, if you intend to maximize creative/generative power available to the user, and you still want people to be communicating with each other, rather than inhabiting solipsistic worlds. You need some common frame so that all the morphing, opening of rabbit holes to new spaces, etc, doesn’t tear the shared virtuality apart, geographically and culturally. You probably also need some kind of rules on who can create and puppet specific personas, so that you can’t have just anyone wearing your face (whether that’s your natural face, or one that you designed for your own use).
They say Kimi K2 is good at writing fiction (Chinese web novels, originally). I wonder if it is specifically good at plot, or narrative causality? And if Eliezer and his crew had serious backing from billionaires, with the correspondingly enhanced ability to develop big plans and carry them out, I wonder if they really would do something like this on the side, in addition to the increasingly political work of stopping frontier AI?
In physics, it is sometimes asked why there should be just three (large) space dimensions. No one really knows, but there are various mathematical properties unique to three or four dimensions, to which appeal is sometimes made.
I would also consider the recent (last few decades) interest in the emergence of spatial dimensions from entanglement. It may be that your question can be answered by considering these two things together.
not the worst outcome
Are you imagining a basically transhumanist future where people have radical longevity and other such boons, but they happen to be trapped within a particular culture (whether that happens to be Christian homeschooling or Bay Area rationalism)? Or could this also be a world where people live lives with a brevity and hazardousness comparable to historic human experience, and in which, in addition, their culture has an unnatural stability maintained by AI working in the background?
It would be interesting to know the extent to which the distribution of beliefs in society is already the result of persuasion. We could then model the immediate future in similar terms, but with the persuasive “pressures” amplified by human-directed AI.
One way to think about it is that progress in AI capabilities means ever bigger and nastier surprises. You find that your AIs can produce realistic but false prose in abundance, you find that they have an inner monologue capable of deciding whether to lie, you find that there are whole communities of people doing what their AIs tell them to do… And humanity has failed if this escalation results in a nasty surprise big enough that it’s fatal for human civilization, that happens before we get to a transhuman world that is nonetheless safe even for mere humans (e.g. Ilya Sutskever’s “plurality of humanity-loving AGIs”).
What are the groups?
Meta is not on that list of “frontier AI” companies because it hasn’t kept up. As far as I know its most advanced model is Llama 4 and that’s not on the same level as GPT-5, Gemini, Grok, or Claude. Not only has it been left behind by the pivot to reasoning models; Meta’s special strength was supposed to be open source, but even there, Chinese models from Moonshot (Kimi K2) and DeepSeek (r2, v3) seem to be ahead. Of course Meta is now trying to get back in the game, but for now they have slipped out of contention.
The remaining question I have concerns the true strength of Chinese AI models, with respect to each other and their American rivals. You could turn my previous paragraph into a thesis about the state of the world: it’s the era of reasoning models, and at the helm are four closed-weight American models and two open-weight Chinese models. But what about Baidu’s Ernie, Alibaba’s Qwen, Zhipu’s ChatGLM? Should they be placed in the first tier as well?
You could be a longtermist and still regard a singleton as the most likely outcome. It would just mean that a human-aligned singleton is the only real chance for a human-aligned long-term future, and so you’d better make that your priority, however unlikely it may be. It’s apparent that a lot of the old-school (pre-LLM) AI-safety people think this way, when they talk about the fate of Earth’s future lightcone and so forth.
However, I’m not familiar with the balance of priorities espoused by actual self-identified longtermists. Do they typically treat a singleton as just a possibility rather than an inevitability?
If I understand correctly, your chief proposition is that liberal rationalists who are shocked and appalled by Trump 2.0 should check out the leftists who actually predicted that Trump 2.0 would be shocking and appalling, rather than just being a new flavor of business as usual. And you hope for adversarial collaboration with a “right-of-center rationalist” who will take the other side of the argument.
The way it’s set up, you seem to want your counterpart to defend the idea that Trump 2.0 is still more business-as-usual, than a disastrous departure from norms. However, there is actually a third point-of-view, that I believe is held by many of those who voted for Trump 2.0.
It was often said of those who voted for Trump 1.0, that they wanted a wrecking-ball—not out of nihilism, but because “desperate times call for desperate measures”. For such people, America was in decline, and the American political class and the elite institutions had become a hermetic world of incompetence and impunity.
For such people—a mix of conservatives and alienated ex-liberals, perhaps—business as usual is the last thing they want. For them, your double crux and forward predictions won’t have the intended diagnostic meaning, because they want comprehensive change, and expect churn and struggle and false starts. They may have very mixed feelings towards Trump and his people, but still prefer the populist and/or nationalist agenda to anything else that’s on offer.
I don’t know if anyone like that will step forward to debate you, but if they do, I’m not sure what the protocol would be.
edit: Maybe the most interesting position would be an e/acc Trump 2.0 supporter—someone from the tech side of Trump’s coalition, rather than the populist side. But such people avoid Less Wrong, I think.
Accelerando (by Stross) presents a model of the singularity which I think can be most fruitfully contrasted with that in A Fire Upon the Deep (by Vinge).
In Accelerando, you even have mind uploads before “the moment of maximum change”. The big change comes when the inner planets begin to be dismantled into swarms of nanocomputers that orbit the sun in the shells of a Dyson structure (a Matrioshka brain). This provides a sudden leap in compute of several orders of magnitude, and implicitly it’s the creation of these vast new virtual spaces, and the migration of 80% of posthuman civilization into those spaces, which finally allows superintelligence to come about.
On the other hand, A Fire Upon the Deep begins with a human expedition poking through an ancient alien data-library. The humans are well aware that there can be dangers in ancient archives, and they think they are just safely browsing, but in fact they unintentionally respawn a malign AI which, when it’s ready, bootstraps its way to superintelligence and kills them all.
In Accelerando, it’s abundance of physical compute which carries transhuman society as a whole beyond human comprehension. In A Fire Upon the Deep, it’s some kind of algorithm which makes the difference—when that algorithm runs, unstoppable superintelligence is created.
Stories are often governed by association rather than logic. In this case, the association is something like “impossible things can’t exist; if you find out that the world is impossible, it will cease to exist”. This motif has occurred before in literature.
It has long been a mystery to me why there isn’t more money in politics
Can you add some context to this remark? I think it’s widely believed that politics is already suffused with money (and that this is a bad thing).
For an analysis of the AI race, this is unusually broad-context and big-picture. There’s a lot to digest.
My main thoughts for now, are that I disagree with two aspects of your historical narrative. I’m not sure they matter that much to the conclusions, but I’ll state them.
First idea that I disagree with: the idea that nuclear proliferation was largely a matter of America deciding who it would share the technology with, on the basis of religious affinity. Once World War II was over, the US didn’t want to share nuclear weapons with anyone, not even Britain. The initial proliferation that occurred was much more a matter of states independently reaching the scientific and industrial level needed to develop nuclear weapons on their own, and America being unable to stop them. The first states across the threshold were disproportionately Judeo-Christian because nuclear knowledge was most advanced in that civilization, but it didn’t take many years for the rest of the world to catch up.
At that point, the five permanent members of the UN Security Council, who had a privileged status within the world system set up after World War II, now had a common interest in curtailing proliferation, and the NNPT was engineered in a way that reflects that. Between the lines, the NNPT says that within the parts of the world that have accepted the treaty, only those five states are allowed to have nuclear weapons. Oh yes, they commit to helping everyone else with civilian nuclear programs, and working towards nuclear disarmament in the long term; but in the short term, the meaning of the NNPT was that only the big five “should” have nuclear weapons. In practice, the big five have gone on to engage in covert acts of proliferation for strategic reasons, as well as developing policies to deal with the reality of nuclear weapons states outside the treaty system, but formally the ideal has been that, in the short term, only the big five should have nukes, and in the long term, no one should.
The idea that the US was an all-powerful gatekeeper of nuclear proliferation that chose to let it happen only within the Judeo-Christian sphere, I think is a perspective distorted by how the world looks now, compared to how it was 80 years ago. Back then, Britain and France weren’t just American allies of the European region, they were at the head of international empires. They conceived themselves to be independent great powers who had made a strategic choice to work with the US… You may have run across the idea that historically, Europe has been united far less often than, say, China. The present situation of a quasi-federal Europe under American tutelage is not the norm, and did not yet exist at the dawn of the nuclear age.
Second idea that I disagree with: the associated idea that during the Cold War, world economic development proceeded along the same lines of civilizational/religious affinity, as in your hypothesis about nuclear proliferation.
I would roughly view the history as follows. Before the industrial revolution, you could indeed view world history in terms of competing cultures, religions, or civilizations. At that time, the whole world was poor by modern standards. Then, the industrial revolution happened within parts of Europe and created a new kind of society with a potential for an unprecedented standard of living.
Some of those societies were thereby empowered to create worldwide empires that combined industrial capitalism at home with old-fashioned imperial resource extraction abroad. From the perspective of competition among civilizations, this looked like the worldwide triumph of Christian European civilization. However, it was really the triumph of a new kind of society and culture (“modernity”) that clashed with many aspects of European tradition (see: Nietzsche and the death of God), and components of which could be mastered by other civilizations (see: Imperial Japan and every decolonization movement).
By the time you get to the cold war and the atomic age, those European empires are already disintegrating, as you now have elites outside the West who wish to do as Japan did, and develop their own version of modernity. Part of the competition between the two new superpowers, America and Russia, was to promote their own social ideology as the key to sovereign economic development. They both tried and failed to be hegemonic within their spheres of influence (China complained of “social imperialism” from Russia, India was non-aligned), but we were not yet fully back in the world where it was an old-fashioned competition between traditional civilizations. It was a transitional period in which two secular universalisms—capitalist democracy and socialist revolution—competed to be the template of modernization.
When Russia abandoned its ideology, there were two interpretations proposed by American political scientists. Fukuyama proposed that the liberal-democratic version of modernity would now be universal. From this perspective, the entire Cold War was a factional fight among Hegelians as to the climactic form of modernity. Hegel had a model of historical progress in the form of broadening empowerment, from absolutism, to aristocracy, and finally to a democratic republic with a concept of citizenship. The Left Hegelian, Marx, proposed one more dialectical stage in the form of communism. The Right Hegelians thought this was a mistake, and Fukuyama declared them the winners.
In the 1990s, this vision looked plausible. Everything was about markets and the spread of democracy. Ideological dissent was limited to small “rogue states”. But in the 2000s, the US found itself fighting jihadis with a very different civilizational vision; in the 2010s, world economic management required giving BRICS a seat at the table alongside the G-7; and in the 2020s, the US is governed by a movement (MAGA) that openly affiliates itself specifically with the West, rather than with the world as a whole. All this conforms to the predictions of Fukuyama’s rival Huntington, who predicted that the postmodern future would see a modernized version of the “clash of civilizations” already seen in the preindustrial past.
Summary. What I’m proposing is that your analytic framework, when it comes to nuclear proliferation and economic development during the Cold War, is the wrong one. To a great extent it does apply within the post-1990s world, but the Cold War was organized around competing secular modernities, not (e.g.) “Christianity versus Atheism”.