Yeah, viatopia is another idea in the same cluster, but it’s explicitly framed as post-ASI:
Yet almost no one has articulated a positive vision for what comes after superintelligence. Few people are even asking, “What if we succeed?” Even fewer have tried to answer.
Same post also says it’s meant to be a generalization of Long Reflection:
Viatopia is a more general concept: the long reflection is one proposal for what viatopia would look like, but it need not be the only one.
However my memory says that the Long Reflection was meant to be pre-ASI, and looking back at where it was originally proposed (Toby Ord’s The Precipice), that still seems like the most plausible interpretation, although it wasn’t fully explicit about this. (Note that the Long Self-Correction is also meant to be pre-ASI, since I think humans are too flawed/unsafe to try to build ASI.) Quoting from Toby’s book (bolding added by me):
How can humanity have the greatest chance of achieving its potential? I think that at the highest level we should adopt a strategy proceeding in three phases:2
Reaching Existential Security
The Long Reflection
Achieving Our Potential
On this view, the first great task for humanity is to reach a place of safety—a place where existential risk is low and stays low. I call this existential security.
It has two strands. Most obviously, we need to preserve humanity’s potential, extracting ourselves from immediate danger so we don’t fail before we’ve got our house in order. This includes direct work on the most pressing existential risks and risk factors, as well as near-term changes to our norms and institutions.
But we also need to protect humanity’s potential—to establish lasting safeguards that will defend humanity from dangers over the longterm future, so that it becomes almost impossible to fail.3 Where preserving our potential is akin to fighting the latest fire, protecting our potential is making changes to ensure that fire will never again pose a serious threat.4 This will involve major changes to our norms and institutions (giving humanity the prudence and patience we need), as well as ways of increasing our general resilience to catastrophe. This needn’t require foreseeing all future risks right now. It is enough if we can set humanity firmly on a course where we will be taking the new risks seriously: managing them successfully right from their onset or sidestepping them entirely.
Note that existential security doesn’t require the risk to be brought down to zero. That would be an impossible target, and attempts to achieve it may well be counter-productive. What humanity needs to do is bring this century’s risk down to a very low level, then keep gradually reducing it from there as the centuries go on. In this way, even though there may always remain some risk in each century, the total risk over our entire future can be kept small.5 We could view this as a form of existential sustainability. Futures in which accumulated existential risk is allowed to climb toward 100 percent are unsustainable. So we need to set a strict risk budget over our entire future, parceling out this non-renewable resource with great care over the generations to come.
Ultimately, existential security is about reducing total existential risk by as many percentage points as possible. Preserving our potential is helping lower the portion of the total risk that we face in the next few decades, while protecting our potential is helping lower the portion that comes over the longer run. We can work on these strands in parallel, devoting some of our efforts to reducing imminent risks and some to building the capacities, institutions, wisdom and will to ensure that future risks are minimal.6
A key insight motivating existential security is that there appear to be no major obstacles to humanity lasting an extremely long time, if only that were a key global priority. As we saw in Chapter 3, we have ample time to protect ourselves against natural risks: even if it took us millennia to resolve the threats from asteroids, supervolcanism and supernovae, we would incur less than one percentage point of total risk.
The greater risk (and tighter deadline) stems from the anthropogenic threats. But being of humanity’s own making, they are also within our control. Were we sufficiently patient, prudent and coordinated, we could simply stop imposing such risks upon ourselves. We would factor in the hidden costs of carbon emissions (or nuclear weapons) and realize they are not a good deal. We would adopt a more mature attitude to the most radical new technologies—devoting at least as much of humanity’s brilliance to forethought and governance as to technological development.
In the past, the survival of humanity didn’t require much conscious effort: our past was brief enough to evade the natural threats and our power too limited to produce anthropogenic threats. But now our longterm survival requires a deliberate choice to survive. As more and more people come to realize this, we can make this choice. There will be great challenges in getting people to look far enough ahead and to see beyond the parochial conflicts of the day. But the logic is clear and the moral arguments powerful. It can be done.
If we achieve existential security, we will have room to breathe. With humanity’s longterm potential secured, we will be past the Precipice, free to contemplate the range of futures that lie open before us. And we will be able to take our time to reflect upon what we truly desire; upon which of these visions for humanity would be the best realization of our potential. We shall call this the Long Reflection.7
Yeah, viatopia is another idea in the same cluster, but it’s explicitly framed as post-ASI:
Same post also says it’s meant to be a generalization of Long Reflection:
However my memory says that the Long Reflection was meant to be pre-ASI, and looking back at where it was originally proposed (Toby Ord’s The Precipice), that still seems like the most plausible interpretation, although it wasn’t fully explicit about this. (Note that the Long Self-Correction is also meant to be pre-ASI, since I think humans are too flawed/unsafe to try to build ASI.) Quoting from Toby’s book (bolding added by me):
@Toby_Ord @wdmacaskill in case they want to weigh in on this.