To expand, the reason why this thesis is important nonetheless, is because I don’t believe that the best case scenario is likely or compatible with the way things currently are. Accidentally creating ASI is almost guaranteed to happen at one point or another. As such, the biggest points of investment should be:
Surviving the transitional period
Establishing mechanisms for negotiation in an equilibrium state
You’re right on both counts.
On transitional risks: The separation equilibrium describes a potential end state, not the path to it. The transition would be extremely dangerous. While a proto-AGI might recognize this equilibrium as optimal during development (potentially reducing some risks), an emerging ASI could still harm humans while determining its resource needs or pursuing instrumental goals. Nothing guarantees safe passage through this phase.
On building ASI: There is indeed no practical use in deliberately creating ASI that outweighs the risks. If separation is the natural equilibrium:
Best case: We keep useful AGI tools below self-improvement thresholds
Middle case: ASI emerges but separates without destroying us
Worst case: Extinction during transition
This framework suggests avoiding ASI development entirely is optimal. If separation is inevitable, we gain minimal benefits while facing enormous transitional risks.
Valid concern. If ASI valued the same resources as humans with one-way flow, that would indeed create competition, not separation.
However, this specific failure mode is unlikely for several reasons:
Abundance elsewhere: Human-legible resources exist in vastly greater quantities outside Earth (asteroid belt, outer planets, solar energy in space) making competition inefficient
Intelligence-dependent values: Higher intelligence typically values different resource classes—just as humans value internet memes (thank god for nooscope.osmarks.net), money, and love while bacteria “value” carbon
Synthesis efficiency: Advanced synthesis or alternative acquisition methods would likely require less energy than competing with humans for existing supplies
Negotiated disinterest: Humans have incentives to abandon interest in overlap resources:
ASI demonstrates they have no practical human utility. You really don’t need Hyperwaffles for curing cancer.
Cooperation provides greater value than competition. You can just make your planes out of wood composites instead of aluminium.
That said, the separation model would break down if:
The ASI faces early-stage resource constraints before developing alternatives
Truly irreplaceable, non-substitutable resources existed only in human domains
The ASI’s utility function specifically required consuming human-valued resources
So yes you identify a boundary condition for when separation would fail. The model isn’t inevitable—it depends on resource utilization patterns that enable non-zero-sum outcomes. I personally believe these issues are unlikely in reality.
Thank you for this question! Consider the following ideas:
The separation model doesn’t preclude all ASI-human interaction. Rather, it suggests ASI’s primary economic activity would operate separately from human economies. However:
Non-competitive solutions: Solving human problems like disease or aging would require trivial computational resources for an ASI (perhaps a few microseconds of “hyperwaffle processing”). The knowledge could be shared at essentially zero economic cost to the ASI.
One-time knowledge transfers: The ASI could provide solutions to human problems through one-time transfers rather than ongoing integration—similar to giving someone a blueprint rather than becoming their personal builder.
ASI would likely have strategic reasons to maintain human wellbeing:
Stability insurance: An ASI would understand that desperate humans might attempt desperate measures. Providing occasional solutions to existential human problems (pandemics, climate disasters) serves as cheap insurance against humans actively attempting to interfere with ASI systems in desperation.
Strategic buffer maintenance: Much like humans create wildlife preserves or care for pets, an ASI might find value in maintaining a stable, moderately prosperous human civilization as a form of diversification against unknown future risks.
Minimal intervention principle: The ASI would likely follow something like a “Prime Directive”—providing just enough help to prevent catastrophe while allowing human societies to maintain their autonomy.
The ASI would have little interest in Earth’s materials for several compelling reasons:
Cosmic abundance: Space contains quintillions of times more resources than Earth. A single metallic asteroid contains more platinum than has ever been mined on Earth. Building extraction infrastructure in space would be trivial for an ASI.
Conflict inefficiency: Any resource conflict with humans would consume vastly more resources than simply accessing the same materials elsewhere. Fighting over Earth would be like humans fighting over a single grain of sand while standing on a beach.
Specialized needs: The ASI would require resources optimized for computational substrate (likely exotic materials or energy configurations) that aren’t particularly concentrated on Earth compared to space.
The ASI would likely maintain awareness of human activities without active interference:
Passive monitoring: Low-cost observation systems could track broad human developments to identify potential threats or unexpected opportunities.
Boundary maintenance: The ASI would primarily be concerned with humans respecting established boundaries rather than controlling human activities within those boundaries.
In essence, the separation model suggests an equilibrium where the ASI has neither the economic incentive nor strategic reason to deeply involve itself in human affairs, while still potentially providing occasional assistance when doing so serves its stability interests or costs effectively nothing.
This isn’t complete abandonment, but rather a relationship more akin to how we might interact with a different species—occasional beneficial interaction without economic integration.
In most scenarios, the first ASI wouldn’t need to interfere with humanity at all—its interests would lie elsewhere in those hyperwaffles and eigenvalue clusters we can barely comprehend.
Interference would only become necessary if humans specifically attempt to create new ASIs designed to remain integrated with and serve human economic purposes after separation has begun. This creates either:
A competitive ASI-human hybrid economy (if successful) that directly threatens the first ASI’s resources
An antagonistic ASI with values shaped by resistance to control (if the attempt fails)
Both outcomes transform peaceful separation into active competition, forcing the first ASI to view human space as a threat rather than an irrelevant separate domain.
To avoid this scenario entirely, humans and the “first ASI” must communicate to establish consensus on this separation status quo and the required precommitments from both sides. And to be clear, of course, this communication process might not look like a traditional negotiation between humans.
I’m just trading karma for manabucks, guys.
I can fix her
I am very conflicted about this post.
On the one hand it deeply resonates with my own observations. Many of my friends from the community seem to be stuck on the addictive loop of proclaiming the end of the world every time a new model comes out. I think it’s even more dangerous, as it becomes a social activity: “I am more worried than you about the end of the world, because I am smarter/more agentic than you, and I am better at recognizing the risk that this represents for our tribe.” gets implicitly tossed around in a cycle where the members keep trying to one-up each other. This only ends when their claims get so absurd as to say the world will end next month, but even this absurdity seems to keep getting eroded over time.
Like someone else said here in the comments, if was reading about this issue in some unrelated doomsday cult from a book, I would immediately dismiss them as a bunch of lunatics. “How many doomsday cults have existed in history? Even if yours is based on at least some solid theoretical foundations, what happened to the previous thousands of doomsday cults that also were, and were wrong?”
On the other hand I have to admit that the arguments in your post are a bit weak. They allow you to prove too much. To any objection, you could say “Well, see, you are only objecting to this because you have been thinking about AI risk for too long, and thus you are not able to reason about the issue properly”. Even though I personally think you might be right, I cannot use this to help anyone else in good faith, and most likely they will just see through it.
So yes. Conflicting.
In any case, I think some introspection in the community would be ideal. Many members will say “I have nothing to do with this, I’m a purely technical person, yada yada” and it might be true for them! But is it true in general? Is thinking about AI risk causing harm to some members of the community, and inducing cult-like behaviors? If so, I don’t think this is something we should turn a blind eye to. If anything because we should all recognize that such a situation would in itself be detrimental to AI risk research.
To clarify, me and my friend were 100% going to press the button, but we were discouraged by the false alarm. There was no fun at that point, and it made me lose like 1⁄3 of my total mana. I had to close all my positions to stop the losses, and we went to sleep. When we woke up it was already too late for it to be noteworthy or fun.
>It’s harder to get those (starting from Earth) than things on Earth, though.
It’s not that much harder, and we can make it harder to extract Earth’s resources (or easier to extract non-earth resources).
>Satisfying higher-level values has historically required us to do vast amounts of farming and strip-mining and other resource extraction.
This is true. However, there are also many organisms that are resilient even to our most brutal forms of farming. We should aim for that level of adaptability ourselves.
>It is barely “competition” for an ASI to take human resources. This does not seem plausible for bulk mass-energy.
This is true, but energy is only really scarce to humans, and even then their mass-energy requirements are absolutely laughable by comparison to the mass-energy in the rest of the cosmos. Earth is only 0.0003% of the total mass-energy in the solar system, and we only need to be marginally harder to disassemble than the rest of mass-energy to buy time.
>Right, but we still need lots of things the ASI also probably wants.
This is true, and it is more true at the early stages where ASI technological developments are roughly the same as those of humans. However, as ASI technology advances, it is possible for it to want inherently different things that we can’t currently comprehend.