I assume Manifold here means “reality”, and not just the betting site?
Mitchell_Porter
Karma: 9,287
I don’t know that this would fit with the idea of no free will. Surely you’re not really making any decisions.
This sounds like “epiphenomenalism”—the idea that the conscious mind has no causal power, it’s just somehow along for the ride of existence, while atoms or whatever do all the work. This is a philosophy that alienates you from your own power to choose.
But there is also “compatibilism”. This is originally the idea that free will is compatible with determinism, because free will is here defined to mean, not that personal decisions have no causes at all, but that all the causes are internal to the person who decides.
A criticism of compatibilism is that this definition isn’t what’s meant by free will. Maybe so. But for the present discussion, it gives us a concept of personal choice which isn’t disconnected from the rest of cause and effect.
We can consider simpler mechanical analogs. Consider any device that “makes choices”, whether it’s a climate control system in a building, or a computer running multiple processes. Does epiphenomenalism make sense here? Is the device irrelevant to the “choice” that happens? I’d say no: the device is the entity that performs the action. The action has a cause, but it is the state of the device itself, along with the relevant physical laws, which is the cause.
We can think similarly of human actions where conscious choice is involved.
But your values wouldn’t have been decided by you.
Perhaps you didn’t choose your original values. But a person’s values can change, and if this was a matter of self-aware choice between two value systems, I’m willing to say that the person decided on their new values.
AI interpretability can assign meaning to states of an AI, but what about process? Are there principled ways of concluding that an AI is thinking, deciding, trying, and so on?
It would hardly be the first time that someone powerful went mad, or was thought to be mad by those around them, and the whole affair was hushed up, or the courtiers just went along with it. Wikipedia says that the story of the emperor’s new clothes goes back at least to 1335… Just last month, Zvi was posting someone’s theory about why rich people go mad. I think the first time I became aware of the brewing alarm around “AI psychosis” was the case of Geoff Lewis, a billionaire VC who has neither disowned his AI-enhanced paranoia of a few months ago, nor kept going with it (instead he got married). And I think I first heard of “vibe physics” in connection with Uber founder Trevor Kalanick.
The consequences for an individual depend on the details. For example, if you still understand yourself as being part of the causal chain of events, because you make decisions that determine your actions—it’s just that your decisions are in turn determined by psychological factors like personality, experience, and intelligence—your sense of agency may remain entirely unaffected. The belief could even impact your decision-making positively, e.g. via a series of thoughts like “my decisions will be determined by my values”—“what do my values actually imply I should do in this situation”—followed by enhanced attention to reasoning about the decision.
On the other hand, one hears that loss of belief in free will can be accompanied by loss of agency or loss of morality, so, the consequences really depend on the psychological details. In general, I think an anti-free-will position that alienates you from the supposed causal machinery of your decision-making, rather than one that identifies you with it, has the potential to diminish a person.
I have three paradigms for how something like this might “work” or at least be popular:
Filters as used in smartphone photos and videos. Here the power to modify the image takes place strictly as an addendum to the context of real human-to-human communication. The Sora 2 app seems a bit like an attempt to apply this model to the much more powerful capabilities of generative video.
The Sora 1 feed. This is just a feed of images and videos created by users, that other users can vote on. The extra twist is that you can usually see the prompt, storyboard, and source material used to generate them, so you can take that material and create your own variations… This paradigm is that of a genuine community of creators—people who were using Sora anyway, and are now able to study and appropriate each other’s creations. One difference between this paradigm and the “filter” paradigm, is that the characters appearing in the creations are not the users, they are basically famous or fictional people.
Virtual reality / shared gaming worlds. It seems to me that something like this is favorable, if you intend to maximize creative/generative power available to the user, and you still want people to be communicating with each other, rather than inhabiting solipsistic worlds. You need some common frame so that all the morphing, opening of rabbit holes to new spaces, etc, doesn’t tear the shared virtuality apart, geographically and culturally. You probably also need some kind of rules on who can create and puppet specific personas, so that you can’t have just anyone wearing your face (whether that’s your natural face, or one that you designed for your own use).
They say Kimi K2 is good at writing fiction (Chinese web novels, originally). I wonder if it is specifically good at plot, or narrative causality? And if Eliezer and his crew had serious backing from billionaires, with the correspondingly enhanced ability to develop big plans and carry them out, I wonder if they really would do something like this on the side, in addition to the increasingly political work of stopping frontier AI?
In physics, it is sometimes asked why there should be just three (large) space dimensions. No one really knows, but there are various mathematical properties unique to three or four dimensions, to which appeal is sometimes made.
I would also consider the recent (last few decades) interest in the emergence of spatial dimensions from entanglement. It may be that your question can be answered by considering these two things together.
not the worst outcome
Are you imagining a basically transhumanist future where people have radical longevity and other such boons, but they happen to be trapped within a particular culture (whether that happens to be Christian homeschooling or Bay Area rationalism)? Or could this also be a world where people live lives with a brevity and hazardousness comparable to historic human experience, and in which, in addition, their culture has an unnatural stability maintained by AI working in the background?
It would be interesting to know the extent to which the distribution of beliefs in society is already the result of persuasion. We could then model the immediate future in similar terms, but with the persuasive “pressures” amplified by human-directed AI.
One way to think about it is that progress in AI capabilities means ever bigger and nastier surprises. You find that your AIs can produce realistic but false prose in abundance, you find that they have an inner monologue capable of deciding whether to lie, you find that there are whole communities of people doing what their AIs tell them to do… And humanity has failed if this escalation results in a nasty surprise big enough that it’s fatal for human civilization, that happens before we get to a transhuman world that is nonetheless safe even for mere humans (e.g. Ilya Sutskever’s “plurality of humanity-loving AGIs”).
What are the groups?
Meta is not on that list of “frontier AI” companies because it hasn’t kept up. As far as I know its most advanced model is Llama 4 and that’s not on the same level as GPT-5, Gemini, Grok, or Claude. Not only has it been left behind by the pivot to reasoning models; Meta’s special strength was supposed to be open source, but even there, Chinese models from Moonshot (Kimi K2) and DeepSeek (r2, v3) seem to be ahead. Of course Meta is now trying to get back in the game, but for now they have slipped out of contention.
The remaining question I have concerns the true strength of Chinese AI models, with respect to each other and their American rivals. You could turn my previous paragraph into a thesis about the state of the world: it’s the era of reasoning models, and at the helm are four closed-weight American models and two open-weight Chinese models. But what about Baidu’s Ernie, Alibaba’s Qwen, Zhipu’s ChatGLM? Should they be placed in the first tier as well?
Understanding the state of frontier AI in China
You could be a longtermist and still regard a singleton as the most likely outcome. It would just mean that a human-aligned singleton is the only real chance for a human-aligned long-term future, and so you’d better make that your priority, however unlikely it may be. It’s apparent that a lot of the old-school (pre-LLM) AI-safety people think this way, when they talk about the fate of Earth’s future lightcone and so forth.
However, I’m not familiar with the balance of priorities espoused by actual self-identified longtermists. Do they typically treat a singleton as just a possibility rather than an inevitability?
If I understand correctly, your chief proposition is that liberal rationalists who are shocked and appalled by Trump 2.0 should check out the leftists who actually predicted that Trump 2.0 would be shocking and appalling, rather than just being a new flavor of business as usual. And you hope for adversarial collaboration with a “right-of-center rationalist” who will take the other side of the argument.
The way it’s set up, you seem to want your counterpart to defend the idea that Trump 2.0 is still more business-as-usual, than a disastrous departure from norms. However, there is actually a third point-of-view, that I believe is held by many of those who voted for Trump 2.0.
It was often said of those who voted for Trump 1.0, that they wanted a wrecking-ball—not out of nihilism, but because “desperate times call for desperate measures”. For such people, America was in decline, and the American political class and the elite institutions had become a hermetic world of incompetence and impunity.
For such people—a mix of conservatives and alienated ex-liberals, perhaps—business as usual is the last thing they want. For them, your double crux and forward predictions won’t have the intended diagnostic meaning, because they want comprehensive change, and expect churn and struggle and false starts. They may have very mixed feelings towards Trump and his people, but still prefer the populist and/or nationalist agenda to anything else that’s on offer.
I don’t know if anyone like that will step forward to debate you, but if they do, I’m not sure what the protocol would be.
edit: Maybe the most interesting position would be an e/acc Trump 2.0 supporter—someone from the tech side of Trump’s coalition, rather than the populist side. But such people avoid Less Wrong, I think.
Accelerando (by Stross) presents a model of the singularity which I think can be most fruitfully contrasted with that in A Fire Upon the Deep (by Vinge).
In Accelerando, you even have mind uploads before “the moment of maximum change”. The big change comes when the inner planets begin to be dismantled into swarms of nanocomputers that orbit the sun in the shells of a Dyson structure (a Matrioshka brain). This provides a sudden leap in compute of several orders of magnitude, and implicitly it’s the creation of these vast new virtual spaces, and the migration of 80% of posthuman civilization into those spaces, which finally allows superintelligence to come about.
On the other hand, A Fire Upon the Deep begins with a human expedition poking through an ancient alien data-library. The humans are well aware that there can be dangers in ancient archives, and they think they are just safely browsing, but in fact they unintentionally respawn a malign AI which, when it’s ready, bootstraps its way to superintelligence and kills them all.
In Accelerando, it’s abundance of physical compute which carries transhuman society as a whole beyond human comprehension. In A Fire Upon the Deep, it’s some kind of algorithm which makes the difference—when that algorithm runs, unstoppable superintelligence is created.
Stories are often governed by association rather than logic. In this case, the association is something like “impossible things can’t exist; if you find out that the world is impossible, it will cease to exist”. This motif has occurred before in literature.
It has long been a mystery to me why there isn’t more money in politics
Can you add some context to this remark? I think it’s widely believed that politics is already suffused with money (and that this is a bad thing).
You have a point in that Vinge portrays outward migration into higher Zones, with all their unexplained advantages including computational advantage, as part of the process by which a civilization of natural intelligences evolves to the point of producing a superintelligence. (For those who haven’t seen the book, the Zones are concentric regions of the galaxy, in which the further out you go, the more advanced the technology that is possible, including superintelligence and faster-than-light travel.)